[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:16:28 dist-test-slave-rq8c krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:16:28 dist-test-slave-rq8c krb5kdc[399](info): set up 2 sockets
Jun 20 09:16:28 dist-test-slave-rq8c krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 09:16:29.442577 392 mini_kdc.cc:121] Time spent starting KDC: real 2.589s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 09:16:29.606431 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.164s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 09:16:29.798770 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.192s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 09:16:29.928939 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.130s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/test-user.keytab.
W20260620 09:16:30.347949 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.419s user 0.000s sys 0.001s
Jun 20 09:16:30 dist-test-slave-rq8c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260620 09:16:31.186527 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.819s user 0.000s sys 0.002s
I20260620 09:16:31.187008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:31.269685 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:31.269901 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:31.269923 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:31.271107 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:31.271153 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:31.271165 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:31.271176 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:31.271188 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:31.272718 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:31.272953 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:31.273187 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:31.278211 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:31.278189 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:31.278287 412 server_base.cc:1034] running on GCE node
W20260620 09:16:31.278196 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:31.278692 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:31.278959 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:31.280113 412 hybrid_clock.cc:648] HybridClock initialized: now 1781946991280095 us; error 34 us; skew 500 ppm
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:31.282671 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:16:31.283154 412 webserver.cc:460] Webserver started at http://127.0.98.62:38025/ using document root <none> and password file <none>
I20260620 09:16:31.283414 412 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:31.283465 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:31.283599 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:31.284711 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "1af0e099ee1c49789aba6856528ee35e"
format_stamp: "Formatted at 2026-06-20 09:16:31 on dist-test-slave-rq8c"
server_key: "514b5f32b73b236b10684659c1fd7438"
server_key_iv: "ee5d042ed0264d3d60e17a97537d2695"
server_key_version: "encryptionkey@0"
I20260620 09:16:31.285104 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "1af0e099ee1c49789aba6856528ee35e"
format_stamp: "Formatted at 2026-06-20 09:16:31 on dist-test-slave-rq8c"
server_key: "514b5f32b73b236b10684659c1fd7438"
server_key_iv: "ee5d042ed0264d3d60e17a97537d2695"
server_key_version: "encryptionkey@0"
I20260620 09:16:31.286545 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:31.287220 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:31.287416 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:31.287504 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "1af0e099ee1c49789aba6856528ee35e"
format_stamp: "Formatted at 2026-06-20 09:16:31 on dist-test-slave-rq8c"
server_key: "514b5f32b73b236b10684659c1fd7438"
server_key_iv: "ee5d042ed0264d3d60e17a97537d2695"
server_key_version: "encryptionkey@0"
I20260620 09:16:31.287570 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:31.310904 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:31.313208 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:31.313364 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:31.317051 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34957
I20260620 09:16:31.317101 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34957 every 8 connection(s)
I20260620 09:16:31.317452 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:31.318320 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:31.320681 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e: Bootstrap starting.
I20260620 09:16:31.321244 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:31.321498 480 log.cc:824] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:31.321857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 412
I20260620 09:16:31.321950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:31.322165 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e: No bootstrap required, opened a new log
I20260620 09:16:31.322561 392 external_mini_cluster.cc:1428] Setting key 7b6175189d1109413a426c73ebd75e12
I20260620 09:16:31.323918 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af0e099ee1c49789aba6856528ee35e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34957 } }
I20260620 09:16:31.324056 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:31.324079 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af0e099ee1c49789aba6856528ee35e, State: Initialized, Role: FOLLOWER
I20260620 09:16:31.324188 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af0e099ee1c49789aba6856528ee35e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34957 } }
I20260620 09:16:31.324261 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:31.324282 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:31.324328 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:31.325127 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af0e099ee1c49789aba6856528ee35e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34957 } }
I20260620 09:16:31.325213 480 leader_election.cc:302] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af0e099ee1c49789aba6856528ee35e; no voters:
I20260620 09:16:31.325363 480 leader_election.cc:288] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:31.325414 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:31.325574 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [term 1 LEADER]: Becoming Leader. State: Replica: 1af0e099ee1c49789aba6856528ee35e, State: Running, Role: LEADER
I20260620 09:16:31.325670 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:31.325696 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af0e099ee1c49789aba6856528ee35e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34957 } }
I20260620 09:16:31.326087 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1af0e099ee1c49789aba6856528ee35e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af0e099ee1c49789aba6856528ee35e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34957 } } }
I20260620 09:16:31.326159 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:31.326371 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1af0e099ee1c49789aba6856528ee35e. Latest consensus state: current_term: 1 leader_uuid: "1af0e099ee1c49789aba6856528ee35e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af0e099ee1c49789aba6856528ee35e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34957 } } }
I20260620 09:16:31.326653 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:31.326803 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:31.327400 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:31.328924 492 catalog_manager.cc:1347] Generated new cluster ID: 625c1abf26b24cebb8d2719f3119dccc
I20260620 09:16:31.328975 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:31.336987 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:31.323354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54146 (local address 127.0.98.62:34957)
0620 09:16:31.323560 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:31.323570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:31.323745 (+ 175us) server_negotiation.cc:407] Connection header received
0620 09:16:31.325505 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:31.325514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:31.325571 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:31.327237 (+ 1666us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:31.327800 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.328932 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.329946 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.330411 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.330506 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:31.333654 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:31.333659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:31.333659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:31.333670 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:31.335186 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:31.335584 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:31.335586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:31.335587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:31.335717 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:31.335953 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:31.335954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:31.335954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:31.336203 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:31.336345 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:31.336617 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 09:16:31.336746 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260620 09:16:31.342406 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:31.342924 492 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:31.353986 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1af0e099ee1c49789aba6856528ee35e: Generated new TSK 0
I20260620 09:16:31.354212 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:16:31.379031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34957
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:31.461313 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:31.461516 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:31.461547 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:31.462671 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:31.462709 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:31.462744 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:31.464229 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:31.464460 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:31.464672 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:31.469403 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:31.469417 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:31.469408 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:31.469596 506 server_base.cc:1034] running on GCE node
I20260620 09:16:31.469775 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:31.470003 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:31.471153 506 hybrid_clock.cc:648] HybridClock initialized: now 1781946991471138 us; error 30 us; skew 500 ppm
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:31.473598 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:16:31.474047 506 webserver.cc:460] Webserver started at http://127.0.98.1:35643/ using document root <none> and password file <none>
I20260620 09:16:31.474256 506 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:31.474304 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:31.474429 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:31.475471 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "688d1fdbb6294d42922eff919f0f2e8d"
format_stamp: "Formatted at 2026-06-20 09:16:31 on dist-test-slave-rq8c"
server_key: "6f06a9753a3564912f05b8c778442628"
server_key_iv: "94496b8686f6cb6f0348c041c292b25d"
server_key_version: "encryptionkey@0"
I20260620 09:16:31.475826 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "688d1fdbb6294d42922eff919f0f2e8d"
format_stamp: "Formatted at 2026-06-20 09:16:31 on dist-test-slave-rq8c"
server_key: "6f06a9753a3564912f05b8c778442628"
server_key_iv: "94496b8686f6cb6f0348c041c292b25d"
server_key_version: "encryptionkey@0"
I20260620 09:16:31.477162 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:31.477917 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:31.478129 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:31.478229 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "688d1fdbb6294d42922eff919f0f2e8d"
format_stamp: "Formatted at 2026-06-20 09:16:31 on dist-test-slave-rq8c"
server_key: "6f06a9753a3564912f05b8c778442628"
server_key_iv: "94496b8686f6cb6f0348c041c292b25d"
server_key_version: "encryptionkey@0"
I20260620 09:16:31.478303 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:31.504324 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:31.506078 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:31.506222 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:31.506448 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:31.506755 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:31.506788 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:31.506819 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:31.506835 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:31.512104 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34139
I20260620 09:16:31.512161 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34139 every 8 connection(s)
I20260620 09:16:31.512446 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:16:31.514029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 506
I20260620 09:16:31.514108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:16:31.514231 392 external_mini_cluster.cc:1428] Setting key 452c835f101f4ebb052f92ed526e0c02
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:16:31.521889 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:31.513083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46539 (local address 127.0.98.62:34957)
0620 09:16:31.513135 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:31.513137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:31.513307 (+ 170us) server_negotiation.cc:407] Connection header received
0620 09:16:31.513795 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:31.513797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:31.513812 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:31.513861 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:31.514918 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.515414 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.516740 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.516945 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.517014 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:31.519380 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:31.519381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:31.519381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:31.519388 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:31.520654 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:31.521065 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:31.521066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:31.521066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:31.521109 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:31.521343 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:31.521343 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:31.521343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:31.521432 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:31.521475 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:31.521768 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 09:16:31.521827 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:16:31.522135 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:31.513050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34957 (local address 127.0.98.1:46539)
0620 09:16:31.513190 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:31.513230 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:31.513702 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:31.513953 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:31.513957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:31.514603 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:31.514837 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:31.514843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.515494 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.515495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:31.516661 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:31.516664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.517033 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.517035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:31.517172 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:31.517648 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:31.517658 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:31.519279 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:31.520728 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:31.520734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:31.520736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:31.520998 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:31.521162 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:31.521163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:31.521163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:31.521287 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:31.521485 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:31.521491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:31.521649 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 09:16:31.521759 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 09:16:31.521908 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260620 09:16:31.522575 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34957
I20260620 09:16:31.522650 635 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:31.522814 635 heartbeater.cc:511] Master 127.0.98.62:34957 requested a full tablet report, sending...
I20260620 09:16:31.523280 444 ts_manager.cc:194] Registered new tserver with Master: 688d1fdbb6294d42922eff919f0f2e8d (127.0.98.1:34139)
I20260620 09:16:31.523939 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:16:31.536613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34957
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:31.625399 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:31.625603 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:31.625636 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:31.626817 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:31.626866 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:31.626911 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:31.628396 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:31.628624 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:31.628825 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:31.633451 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:31.633442 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:31.633423 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:31.633637 640 server_base.cc:1034] running on GCE node
I20260620 09:16:31.633812 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:31.634013 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:31.635138 640 hybrid_clock.cc:648] HybridClock initialized: now 1781946991635126 us; error 30 us; skew 500 ppm
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:31.637459 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:16:31.637910 640 webserver.cc:460] Webserver started at http://127.0.98.2:40285/ using document root <none> and password file <none>
I20260620 09:16:31.638110 640 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:31.638157 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:31.638266 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:31.639298 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "2a4421da2ec34824980ecd669250013a"
format_stamp: "Formatted at 2026-06-20 09:16:31 on dist-test-slave-rq8c"
server_key: "5b3710544ad9a2f5497590cded3e0c23"
server_key_iv: "3d864c335c8fabc40289c235d14ae0c3"
server_key_version: "encryptionkey@0"
I20260620 09:16:31.639667 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2a4421da2ec34824980ecd669250013a"
format_stamp: "Formatted at 2026-06-20 09:16:31 on dist-test-slave-rq8c"
server_key: "5b3710544ad9a2f5497590cded3e0c23"
server_key_iv: "3d864c335c8fabc40289c235d14ae0c3"
server_key_version: "encryptionkey@0"
I20260620 09:16:31.640985 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:31.641706 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:31.641901 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:31.641984 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "2a4421da2ec34824980ecd669250013a"
format_stamp: "Formatted at 2026-06-20 09:16:31 on dist-test-slave-rq8c"
server_key: "5b3710544ad9a2f5497590cded3e0c23"
server_key_iv: "3d864c335c8fabc40289c235d14ae0c3"
server_key_version: "encryptionkey@0"
I20260620 09:16:31.642045 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:31.651660 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:31.653700 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:31.653831 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:31.654033 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:31.654316 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:31.654348 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:31.654377 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:31.654400 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:31.660458 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44191
I20260620 09:16:31.660524 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44191 every 8 connection(s)
I20260620 09:16:31.660811 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:31.668905 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:31.661464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43235 (local address 127.0.98.62:34957)
0620 09:16:31.661509 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:31.661511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:31.661702 (+ 191us) server_negotiation.cc:407] Connection header received
0620 09:16:31.662155 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:31.662156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:31.662165 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:31.662222 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:31.663097 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.663578 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.664532 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.664721 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.664773 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:31.666734 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:31.666736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:31.666736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:31.666743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:31.667842 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:31.668218 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:31.668218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:31.668219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:31.668261 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:31.668447 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:31.668447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:31.668447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:31.668536 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:31.668578 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:31.668783 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 09:16:31.668839 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:16:31.669122 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:31.661441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34957 (local address 127.0.98.2:43235)
0620 09:16:31.661587 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:31.661628 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:31.662082 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:31.662300 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:31.662303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:31.662842 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:31.663027 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:31.663031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.663673 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.663673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:31.664465 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:31.664467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.664784 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.664785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:31.664895 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:31.665310 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:31.665318 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:31.666648 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:31.667912 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:31.667918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:31.667919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:31.668149 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:31.668308 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:31.668308 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:31.668309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:31.668400 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:31.668580 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:31.668582 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:31.668690 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 09:16:31.668769 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 09:16:31.668895 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260620 09:16:31.669590 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34957
I20260620 09:16:31.669682 769 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:31.669865 769 heartbeater.cc:511] Master 127.0.98.62:34957 requested a full tablet report, sending...
I20260620 09:16:31.670249 444 ts_manager.cc:194] Registered new tserver with Master: 2a4421da2ec34824980ecd669250013a (127.0.98.2:44191)
I20260620 09:16:31.670603 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43235
I20260620 09:16:31.670856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 640
I20260620 09:16:31.670944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:16:31.671105 392 external_mini_cluster.cc:1428] Setting key 711d3a7e60f388df635fbae7c7142609
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:16:31.691931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34957
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:31.769765 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:31.769969 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:31.770000 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:31.771139 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:31.771181 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:31.771217 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:31.772671 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:31.772912 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:31.773130 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:31.778046 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:31.778087 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:31.778077 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:31.778268 774 server_base.cc:1034] running on GCE node
I20260620 09:16:31.778442 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:31.778672 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:31.779803 774 hybrid_clock.cc:648] HybridClock initialized: now 1781946991779791 us; error 29 us; skew 500 ppm
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:31.782208 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:16:31.782657 774 webserver.cc:460] Webserver started at http://127.0.98.3:40095/ using document root <none> and password file <none>
I20260620 09:16:31.782882 774 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:31.782929 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:31.783051 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:31.784149 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "ed02ecc0d14d4999bce7e0c34a9db991"
format_stamp: "Formatted at 2026-06-20 09:16:31 on dist-test-slave-rq8c"
server_key: "6e17368e8fc8d0f0acbcd21f976ae34c"
server_key_iv: "a8224bb6d4aec2f070b0b32b08fb9105"
server_key_version: "encryptionkey@0"
I20260620 09:16:31.784463 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ed02ecc0d14d4999bce7e0c34a9db991"
format_stamp: "Formatted at 2026-06-20 09:16:31 on dist-test-slave-rq8c"
server_key: "6e17368e8fc8d0f0acbcd21f976ae34c"
server_key_iv: "a8224bb6d4aec2f070b0b32b08fb9105"
server_key_version: "encryptionkey@0"
I20260620 09:16:31.785662 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:31.786301 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:31.786444 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:31.786523 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "ed02ecc0d14d4999bce7e0c34a9db991"
format_stamp: "Formatted at 2026-06-20 09:16:31 on dist-test-slave-rq8c"
server_key: "6e17368e8fc8d0f0acbcd21f976ae34c"
server_key_iv: "a8224bb6d4aec2f070b0b32b08fb9105"
server_key_version: "encryptionkey@0"
I20260620 09:16:31.786580 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:31.811058 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:31.812966 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:31.813102 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:31.813308 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:31.813589 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:31.813625 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:31.813653 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:31.813675 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:31.819994 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38209
I20260620 09:16:31.820044 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38209 every 8 connection(s)
I20260620 09:16:31.820339 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:31.826242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 774
I20260620 09:16:31.826337 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:16:31.826503 392 external_mini_cluster.cc:1428] Setting key 443d1ca4a5e2fada8696f835bd40c966
I20260620 09:16:31.829321 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:31.821003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45783 (local address 127.0.98.62:34957)
0620 09:16:31.821056 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:31.821058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:31.821262 (+ 204us) server_negotiation.cc:407] Connection header received
0620 09:16:31.821718 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:31.821719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:31.821728 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:31.821774 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:31.822674 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.823156 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.824108 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.824295 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.824365 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:31.826525 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:31.826526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:31.826526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:31.826533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:31.828075 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:31.828419 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:31.828420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:31.828420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:31.828470 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:31.828688 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:31.828695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:31.828695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:31.828817 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:31.828889 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:31.829185 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 09:16:31.829255 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:16:31.829469 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:31.820983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34957 (local address 127.0.98.3:45783)
0620 09:16:31.821149 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:31.821183 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:31.821644 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:31.821837 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:31.821841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:31.822398 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:31.822597 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:31.822602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.823223 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.823223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:31.824047 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:31.824049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.824351 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.824352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:31.824444 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:31.824887 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:31.824894 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:31.826429 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:31.828141 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:31.828148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:31.828149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:31.828359 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:31.828517 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:31.828517 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:31.828518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:31.828613 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:31.828903 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:31.828908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:31.829035 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 09:16:31.829132 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 09:16:31.829255 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260620 09:16:31.829892 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34957
I20260620 09:16:31.829993 903 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:31.830173 903 heartbeater.cc:511] Master 127.0.98.62:34957 requested a full tablet report, sending...
I20260620 09:16:31.830523 444 ts_manager.cc:194] Registered new tserver with Master: ed02ecc0d14d4999bce7e0c34a9db991 (127.0.98.3:38209)
I20260620 09:16:31.830868 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45783
I20260620 09:16:31.839102 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:16:31.861758 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:31.853992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38264 (local address 127.0.98.1:34139)
0620 09:16:31.854173 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:31.854178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:31.854188 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:16:31.854215 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:31.854217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:31.854230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:31.854294 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:31.854802 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.855567 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.856475 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.856810 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.856870 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:31.859057 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:31.859067 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:31.859067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:31.859074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:31.860640 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:31.860946 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:31.860949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:31.860950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:31.861000 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:31.861192 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:31.861194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:31.861194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:31.861347 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:31.861477 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:31.861577 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 09:16:31.861649 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
W20260620 09:16:31.862100 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38264
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:16:31.883329 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:31.876190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.0.98.1:34139)
0620 09:16:31.876249 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:31.876251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:31.876729 (+ 478us) server_negotiation.cc:407] Connection header received
0620 09:16:31.876783 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:31.876784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:31.876798 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:31.876850 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:31.877344 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.877843 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.878587 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:31.878776 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:31.878818 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:31.881045 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:31.881046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:31.881047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:31.881054 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:31.882387 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:31.882719 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:31.882720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:31.882720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:31.882765 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:31.882967 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:31.882967 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:31.882967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:31.883051 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:31.883133 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:31.883198 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 09:16:31.883260 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:16:31.883883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 506
I20260620 09:16:31.888491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 640
I20260620 09:16:31.893738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 774
I20260620 09:16:31.897529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 412
2026-06-20T09:16:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5068 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[917](info): set up 2 sockets
Jun 20 09:16:31 dist-test-slave-rq8c krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 09:16:32.842970 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:16:32 dist-test-slave-rq8c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:16:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:16:32.949558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46865
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:33.026952 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:33.027161 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:33.027182 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:33.028638 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:33.028680 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:33.028694 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:33.028705 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:33.028716 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:33.030109 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:33.030303 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:33.030475 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:33.035008 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:33.035003 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:33.035003 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:33.035523 930 server_base.cc:1034] running on GCE node
I20260620 09:16:33.035758 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:33.036006 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:33.037137 930 hybrid_clock.cc:648] HybridClock initialized: now 1781946993037123 us; error 28 us; skew 500 ppm
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:33.039381 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:16:33.039811 930 webserver.cc:460] Webserver started at http://127.0.98.62:41591/ using document root <none> and password file <none>
I20260620 09:16:33.040010 930 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:33.040052 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:33.040167 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:33.041169 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "3df22135adcf46f3a1fed2e8ef8c707b"
format_stamp: "Formatted at 2026-06-20 09:16:33 on dist-test-slave-rq8c"
server_key: "137dcef907f5bb44904cba994ed79e37"
server_key_iv: "fc9748123edc7270c292129da3647234"
server_key_version: "encryptionkey@0"
I20260620 09:16:33.041496 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "3df22135adcf46f3a1fed2e8ef8c707b"
format_stamp: "Formatted at 2026-06-20 09:16:33 on dist-test-slave-rq8c"
server_key: "137dcef907f5bb44904cba994ed79e37"
server_key_iv: "fc9748123edc7270c292129da3647234"
server_key_version: "encryptionkey@0"
I20260620 09:16:33.042656 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:33.043242 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:33.043403 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:33.043485 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "3df22135adcf46f3a1fed2e8ef8c707b"
format_stamp: "Formatted at 2026-06-20 09:16:33 on dist-test-slave-rq8c"
server_key: "137dcef907f5bb44904cba994ed79e37"
server_key_iv: "fc9748123edc7270c292129da3647234"
server_key_version: "encryptionkey@0"
I20260620 09:16:33.043540 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:33.062240 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:33.064373 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:33.064478 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:33.067994 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46865
I20260620 09:16:33.068028 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46865 every 8 connection(s)
I20260620 09:16:33.068333 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:33.069128 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:33.071280 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b: Bootstrap starting.
I20260620 09:16:33.071835 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:33.072090 998 log.cc:824] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:33.072677 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b: No bootstrap required, opened a new log
I20260620 09:16:33.073874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 930
I20260620 09:16:33.073948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:33.074090 392 external_mini_cluster.cc:1428] Setting key 3957e4d32ddf916eba6690b364fdb41d
I20260620 09:16:33.074281 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df22135adcf46f3a1fed2e8ef8c707b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46865 } }
I20260620 09:16:33.074419 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:33.074450 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df22135adcf46f3a1fed2e8ef8c707b, State: Initialized, Role: FOLLOWER
I20260620 09:16:33.074606 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df22135adcf46f3a1fed2e8ef8c707b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46865 } }
I20260620 09:16:33.074717 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:33.074756 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:33.074800 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:33.075595 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df22135adcf46f3a1fed2e8ef8c707b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46865 } }
I20260620 09:16:33.075721 998 leader_election.cc:302] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3df22135adcf46f3a1fed2e8ef8c707b; no voters:
I20260620 09:16:33.075878 998 leader_election.cc:288] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:33.075912 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:33.076011 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [term 1 LEADER]: Becoming Leader. State: Replica: 3df22135adcf46f3a1fed2e8ef8c707b, State: Running, Role: LEADER
I20260620 09:16:33.076089 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df22135adcf46f3a1fed2e8ef8c707b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46865 } }
I20260620 09:16:33.076121 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:33.076404 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3df22135adcf46f3a1fed2e8ef8c707b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df22135adcf46f3a1fed2e8ef8c707b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46865 } } }
I20260620 09:16:33.076558 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:33.076721 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3df22135adcf46f3a1fed2e8ef8c707b. Latest consensus state: current_term: 1 leader_uuid: "3df22135adcf46f3a1fed2e8ef8c707b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df22135adcf46f3a1fed2e8ef8c707b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46865 } } }
I20260620 09:16:33.076817 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:33.077157 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:33.077718 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:33.079080 1011 catalog_manager.cc:1347] Generated new cluster ID: 812d6d2cf0494ada9274e32ea7c22dd1
I20260620 09:16:33.079128 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:33.087608 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.074559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.0.98.62:46865)
0620 09:16:33.074743 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.074752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.074784 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:16:33.074899 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.074905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.074953 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.075067 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:33.075672 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.076983 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.078118 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.078586 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.078696 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.084271 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:33.084275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:33.084275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:33.084286 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:33.086091 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.086470 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.086472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.086473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.086566 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.086744 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.086745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.086745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.087030 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:33.087173 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.087260 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.087384 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260620 09:16:33.091436 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:33.092139 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:33.100893 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3df22135adcf46f3a1fed2e8ef8c707b: Generated new TSK 0
I20260620 09:16:33.101082 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:16:33.123154 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46865
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:33.200567 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:33.200771 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:33.200802 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:33.201967 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:33.202005 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:33.202040 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:33.203420 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46865
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:33.203668 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:33.203869 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:33.208513 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:33.208541 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:33.208639 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:33.208920 1024 server_base.cc:1034] running on GCE node
I20260620 09:16:33.209129 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:33.209398 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:33.210532 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781946993210522 us; error 29 us; skew 500 ppm
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:33.212991 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:16:33.213413 1024 webserver.cc:460] Webserver started at http://127.0.98.1:34355/ using document root <none> and password file <none>
I20260620 09:16:33.213614 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:33.213657 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:33.213774 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:33.214823 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "cfeba1a35d2948dba0fc684225eb0903"
format_stamp: "Formatted at 2026-06-20 09:16:33 on dist-test-slave-rq8c"
server_key: "5afca6eab50321ff701e4e3d80cf6d4c"
server_key_iv: "58d3a890ed56a012282b733d01136aca"
server_key_version: "encryptionkey@0"
I20260620 09:16:33.215149 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cfeba1a35d2948dba0fc684225eb0903"
format_stamp: "Formatted at 2026-06-20 09:16:33 on dist-test-slave-rq8c"
server_key: "5afca6eab50321ff701e4e3d80cf6d4c"
server_key_iv: "58d3a890ed56a012282b733d01136aca"
server_key_version: "encryptionkey@0"
I20260620 09:16:33.216468 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:33.217175 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:33.217316 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:33.217409 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "cfeba1a35d2948dba0fc684225eb0903"
format_stamp: "Formatted at 2026-06-20 09:16:33 on dist-test-slave-rq8c"
server_key: "5afca6eab50321ff701e4e3d80cf6d4c"
server_key_iv: "58d3a890ed56a012282b733d01136aca"
server_key_version: "encryptionkey@0"
I20260620 09:16:33.217474 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:33.231200 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:33.232991 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:33.233109 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:33.233312 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:33.233596 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:33.233628 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:33.233649 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:33.233672 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:33.239008 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37339
I20260620 09:16:33.239073 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37339 every 8 connection(s)
I20260620 09:16:33.239347 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:33.247679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 1024
I20260620 09:16:33.247754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:16:33.247895 392 external_mini_cluster.cc:1428] Setting key 70d68cc09f290bd55a346417aae54766
I20260620 09:16:33.247956 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.240042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56033 (local address 127.0.98.62:46865)
0620 09:16:33.240099 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.240101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.240272 (+ 171us) server_negotiation.cc:407] Connection header received
0620 09:16:33.240783 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.240785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.240797 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.240849 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:33.241800 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.242306 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.243264 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.243473 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.243528 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.245566 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:33.245568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:33.245569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:33.245580 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:33.246748 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.247112 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.247113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.247113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.247162 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.247355 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.247355 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.247355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.247455 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:33.247498 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.247778 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.247858 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:16:33.248100 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.240001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46865 (local address 127.0.98.1:56033)
0620 09:16:33.240148 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:33.240191 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:33.240684 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:33.240921 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:33.240925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:33.241507 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:33.241700 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.241705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.242395 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.242396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.243177 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.243179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.243539 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.243540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.243706 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.244198 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:33.244207 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:33.245473 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:33.246820 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:33.246826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:33.246827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:33.247052 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:33.247219 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:33.247220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:33.247220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:33.247316 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:33.247495 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:33.247498 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:33.247648 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 09:16:33.247745 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 09:16:33.247895 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260620 09:16:33.248528 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46865
I20260620 09:16:33.248617 1153 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:33.248778 1153 heartbeater.cc:511] Master 127.0.98.62:46865 requested a full tablet report, sending...
I20260620 09:16:33.249279 961 ts_manager.cc:194] Registered new tserver with Master: cfeba1a35d2948dba0fc684225eb0903 (127.0.98.1:37339)
I20260620 09:16:33.250010 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56033
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:16:33.269873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46865
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:33.347992 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:33.348202 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:33.348235 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:33.349404 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:33.349445 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:33.349481 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:33.350894 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46865
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:33.351114 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:33.351318 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:33.355945 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:33.355970 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:33.355926 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:33.356027 1158 server_base.cc:1034] running on GCE node
I20260620 09:16:33.356308 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:33.356496 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:33.357633 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781946993357620 us; error 30 us; skew 500 ppm
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:33.359892 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:16:33.360322 1158 webserver.cc:460] Webserver started at http://127.0.98.2:32779/ using document root <none> and password file <none>
I20260620 09:16:33.360510 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:33.360555 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:33.360674 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:33.361645 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "2eb350fb30944cbdbd14c9f93898b7cb"
format_stamp: "Formatted at 2026-06-20 09:16:33 on dist-test-slave-rq8c"
server_key: "0e249a21be124ad0f58176f57d490d81"
server_key_iv: "c369f48f6b12f73dfc2a7329085794fb"
server_key_version: "encryptionkey@0"
I20260620 09:16:33.361958 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2eb350fb30944cbdbd14c9f93898b7cb"
format_stamp: "Formatted at 2026-06-20 09:16:33 on dist-test-slave-rq8c"
server_key: "0e249a21be124ad0f58176f57d490d81"
server_key_iv: "c369f48f6b12f73dfc2a7329085794fb"
server_key_version: "encryptionkey@0"
I20260620 09:16:33.363092 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:33.363759 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:33.363891 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:33.363968 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "2eb350fb30944cbdbd14c9f93898b7cb"
format_stamp: "Formatted at 2026-06-20 09:16:33 on dist-test-slave-rq8c"
server_key: "0e249a21be124ad0f58176f57d490d81"
server_key_iv: "c369f48f6b12f73dfc2a7329085794fb"
server_key_version: "encryptionkey@0"
I20260620 09:16:33.364037 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:33.386528 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:33.388688 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:33.388823 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:33.389032 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:33.389326 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:33.389358 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:33.389387 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:33.389408 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:33.394701 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35913
I20260620 09:16:33.394763 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35913 every 8 connection(s)
I20260620 09:16:33.395035 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:33.403643 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.395707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39655 (local address 127.0.98.62:46865)
0620 09:16:33.395765 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.395767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.395968 (+ 201us) server_negotiation.cc:407] Connection header received
0620 09:16:33.396454 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.396455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.396468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.396519 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:33.397424 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.397916 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.399023 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.399229 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.399285 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.401270 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:33.401271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:33.401271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:33.401279 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:33.402422 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.402762 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.402763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.402764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.402815 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.403022 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.403023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.403023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.403133 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:33.403199 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.403449 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.403533 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:16:33.403815 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.395696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46865 (local address 127.0.98.2:39655)
0620 09:16:33.395857 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:33.395892 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:33.396362 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:33.396588 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:33.396591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:33.397136 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:33.397329 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.397334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.397997 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.397998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.398935 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.398938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.399295 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.399296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.399421 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.399920 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:33.399928 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:33.401183 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:33.402492 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:33.402498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:33.402500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:33.402698 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:33.402871 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:33.402871 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:33.402872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:33.402969 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:33.403197 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:33.403203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:33.403333 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 09:16:33.403424 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 09:16:33.403571 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260620 09:16:33.404137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 1158
I20260620 09:16:33.404243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:16:33.404278 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46865
I20260620 09:16:33.404374 1287 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:33.404413 392 external_mini_cluster.cc:1428] Setting key 240eb00b943860fadfab5cdf576327ab
I20260620 09:16:33.404547 1287 heartbeater.cc:511] Master 127.0.98.62:46865 requested a full tablet report, sending...
I20260620 09:16:33.404892 962 ts_manager.cc:194] Registered new tserver with Master: 2eb350fb30944cbdbd14c9f93898b7cb (127.0.98.2:35913)
I20260620 09:16:33.405339 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39655
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:16:33.426115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46865
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:33.503968 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:33.504180 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:33.504210 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:33.505318 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:33.505352 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:33.505376 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:33.506707 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46865
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:33.506883 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:33.507053 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:33.511534 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:33.511534 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:33.511603 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:33.511746 1292 server_base.cc:1034] running on GCE node
I20260620 09:16:33.511937 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:33.512135 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:33.513288 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781946993513265 us; error 39 us; skew 500 ppm
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:33.515584 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:16:33.515976 1292 webserver.cc:460] Webserver started at http://127.0.98.3:43997/ using document root <none> and password file <none>
I20260620 09:16:33.516146 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:33.516184 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:33.516276 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:33.517668 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "71ba7aec4edd4515a5fa91755d6201b2"
format_stamp: "Formatted at 2026-06-20 09:16:33 on dist-test-slave-rq8c"
server_key: "65ff26d66a7662407ff70bc6338a2ac0"
server_key_iv: "c32ddcfc3c76343f54eac337f679c2d8"
server_key_version: "encryptionkey@0"
I20260620 09:16:33.517952 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "71ba7aec4edd4515a5fa91755d6201b2"
format_stamp: "Formatted at 2026-06-20 09:16:33 on dist-test-slave-rq8c"
server_key: "65ff26d66a7662407ff70bc6338a2ac0"
server_key_iv: "c32ddcfc3c76343f54eac337f679c2d8"
server_key_version: "encryptionkey@0"
I20260620 09:16:33.519136 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:16:33.519817 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:33.519961 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:33.520037 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "71ba7aec4edd4515a5fa91755d6201b2"
format_stamp: "Formatted at 2026-06-20 09:16:33 on dist-test-slave-rq8c"
server_key: "65ff26d66a7662407ff70bc6338a2ac0"
server_key_iv: "c32ddcfc3c76343f54eac337f679c2d8"
server_key_version: "encryptionkey@0"
I20260620 09:16:33.520109 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:33.532248 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:33.534093 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:33.534228 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:33.534425 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:33.534690 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:33.534723 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:33.534742 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:33.534754 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:33.540024 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41899
I20260620 09:16:33.540082 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41899 every 8 connection(s)
I20260620 09:16:33.540362 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:33.548722 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.541017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35331 (local address 127.0.98.62:46865)
0620 09:16:33.541092 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.541094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.541217 (+ 123us) server_negotiation.cc:407] Connection header received
0620 09:16:33.541739 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.541740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.541752 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.541799 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:33.542697 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.543184 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.544191 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.544390 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.544445 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.546519 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:33.546521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:33.546521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:33.546528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:33.547684 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.548026 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.548027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.548027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.548073 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.548244 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.548245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.548245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.548352 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:33.548396 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.548601 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.548657 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:16:33.548883 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.540942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46865 (local address 127.0.98.3:35331)
0620 09:16:33.541101 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:33.541145 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:33.541635 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:33.541880 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:33.541883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:33.542419 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:33.542607 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.542612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.543279 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.543280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.544114 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.544116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.544454 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.544455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.544568 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.545060 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:33.545068 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:33.546434 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:33.547755 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:33.547760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:33.547762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:33.547968 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:33.548115 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:33.548115 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:33.548116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:33.548204 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:33.548393 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:33.548396 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:33.548507 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 09:16:33.548583 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:16:33.548697 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260620 09:16:33.549283 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46865
I20260620 09:16:33.549384 1421 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:33.549544 1421 heartbeater.cc:511] Master 127.0.98.62:46865 requested a full tablet report, sending...
I20260620 09:16:33.549865 962 ts_manager.cc:194] Registered new tserver with Master: 71ba7aec4edd4515a5fa91755d6201b2 (127.0.98.3:41899)
I20260620 09:16:33.550139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 1292
I20260620 09:16:33.550227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:16:33.550311 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35331
I20260620 09:16:33.550391 392 external_mini_cluster.cc:1428] Setting key 4fd50cfc405c486a55dd21ec19a000ea
I20260620 09:16:33.551116 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:16:33.557610 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.552600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.0.98.62:46865)
0620 09:16:33.552643 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.552645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.552837 (+ 192us) server_negotiation.cc:407] Connection header received
0620 09:16:33.552880 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.552880 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.552888 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.552931 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:33.553399 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.553850 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.554489 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.554660 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.554700 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.555763 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:33.555764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:33.555764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:33.555770 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:33.556817 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.557080 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.557080 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.557080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.557123 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.557229 (+ 106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.557229 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.557229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.557348 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:33.557414 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.557499 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.557552 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:16:33.559609 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:33.561147 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:33.567004 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.563281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.0.98.1:37339)
0620 09:16:33.563515 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.563520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.563531 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:16:33.563568 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.563570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.563585 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.563694 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:33.563824 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.564665 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.566043 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.566639 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.566701 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.566758 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.566886 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.566926 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":70,"threads_started":1}
I20260620 09:16:33.567000 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.563143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37339 (local address 127.0.0.1:54590)
0620 09:16:33.563310 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:33.563318 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:33.563357 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:33.563710 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:33.563710 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:33.563712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:33.563762 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.563764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.564751 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.564752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.565952 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.565955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.566710 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.566711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.566794 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.566807 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 09:16:33.566866 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:16:33.566916 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":32,"threads_started":1}
I20260620 09:16:33.567154 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.563003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35913 (local address 127.0.0.1:50414)
0620 09:16:33.563116 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:33.563156 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:33.563222 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:33.563641 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:33.563644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:33.563646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:33.563711 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.563713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.565065 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.565066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.566087 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.566088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.566913 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.566914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.566977 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.566983 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:16:33.567011 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 09:16:33.567053 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":2,"thread_start_us":42,"threads_started":1}
I20260620 09:16:33.567157 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.563243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41899 (local address 127.0.0.1:59750)
0620 09:16:33.563397 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:33.563405 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:33.563444 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:33.563756 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:33.563757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:33.563758 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:33.563785 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.563786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.565130 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.565131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.565965 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.565967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.566912 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.566913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.566977 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.566982 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:16:33.567011 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 09:16:33.567053 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":14,"thread_start_us":106,"threads_started":1}
I20260620 09:16:33.567397 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.563451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59750 (local address 127.0.98.3:41899)
0620 09:16:33.563591 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.563595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.563603 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:33.563657 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.563658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.563671 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.563861 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:33.564052 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.565056 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.566062 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.566812 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.567183 (+ 371us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.567249 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.567275 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.567314 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260620 09:16:33.567466 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.563106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.0.98.2:35913)
0620 09:16:33.563465 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.563470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.563480 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:16:33.563519 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.563521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.563536 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.563609 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:33.563843 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.564972 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.566147 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.566821 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.567254 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.567313 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.567335 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.567398 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":101,"threads_started":1}
I20260620 09:16:33.568055 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 1d474f5cc2444c9bb9b553cb1d9c7dd6 (DEFAULT_TABLE table=test-table [id=c005453dd4de4a8f864fefe8fb9b2130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:33.568310 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 1d474f5cc2444c9bb9b553cb1d9c7dd6 (DEFAULT_TABLE table=test-table [id=c005453dd4de4a8f864fefe8fb9b2130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:33.568424 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d474f5cc2444c9bb9b553cb1d9c7dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:33.568591 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d474f5cc2444c9bb9b553cb1d9c7dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:33.571902 1435 tablet_bootstrap.cc:492] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903: Bootstrap starting.
I20260620 09:16:33.572175 1436 tablet_bootstrap.cc:492] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2: Bootstrap starting.
I20260620 09:16:33.572329 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 1d474f5cc2444c9bb9b553cb1d9c7dd6 (DEFAULT_TABLE table=test-table [id=c005453dd4de4a8f864fefe8fb9b2130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:33.572585 1435 tablet_bootstrap.cc:654] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:33.572670 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d474f5cc2444c9bb9b553cb1d9c7dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:33.572983 1436 tablet_bootstrap.cc:654] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:33.573174 1435 log.cc:824] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:33.573450 1436 log.cc:824] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:33.573936 1435 tablet_bootstrap.cc:492] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903: No bootstrap required, opened a new log
I20260620 09:16:33.574012 1435 ts_tablet_manager.cc:1397] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:16:33.574066 1436 tablet_bootstrap.cc:492] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2: No bootstrap required, opened a new log
I20260620 09:16:33.574129 1436 ts_tablet_manager.cc:1397] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:33.575449 1436 raft_consensus.cc:348] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35913 } } peers { permanent_uuid: "cfeba1a35d2948dba0fc684225eb0903" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37339 } } peers { permanent_uuid: "71ba7aec4edd4515a5fa91755d6201b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41899 } }
I20260620 09:16:33.575585 1436 raft_consensus.cc:374] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:33.575639 1436 raft_consensus.cc:729] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ba7aec4edd4515a5fa91755d6201b2, State: Initialized, Role: FOLLOWER
I20260620 09:16:33.575802 1436 consensus_queue.cc:260] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35913 } } peers { permanent_uuid: "cfeba1a35d2948dba0fc684225eb0903" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37339 } } peers { permanent_uuid: "71ba7aec4edd4515a5fa91755d6201b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41899 } }
I20260620 09:16:33.576063 1436 ts_tablet_manager.cc:1428] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 09:16:33.575973 1435 raft_consensus.cc:348] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35913 } } peers { permanent_uuid: "cfeba1a35d2948dba0fc684225eb0903" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37339 } } peers { permanent_uuid: "71ba7aec4edd4515a5fa91755d6201b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41899 } }
I20260620 09:16:33.576220 1421 heartbeater.cc:503] Master 127.0.98.62:46865 was elected leader, sending a full tablet report...
I20260620 09:16:33.576172 1435 raft_consensus.cc:374] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:33.576261 1435 raft_consensus.cc:729] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfeba1a35d2948dba0fc684225eb0903, State: Initialized, Role: FOLLOWER
I20260620 09:16:33.576411 1435 consensus_queue.cc:260] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35913 } } peers { permanent_uuid: "cfeba1a35d2948dba0fc684225eb0903" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37339 } } peers { permanent_uuid: "71ba7aec4edd4515a5fa91755d6201b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41899 } }
I20260620 09:16:33.576462 1440 tablet_bootstrap.cc:492] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb: Bootstrap starting.
I20260620 09:16:33.576709 1435 ts_tablet_manager.cc:1428] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:16:33.576718 1153 heartbeater.cc:503] Master 127.0.98.62:46865 was elected leader, sending a full tablet report...
I20260620 09:16:33.577410 1440 tablet_bootstrap.cc:654] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:33.577922 1440 log.cc:824] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:33.578584 1440 tablet_bootstrap.cc:492] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb: No bootstrap required, opened a new log
I20260620 09:16:33.578639 1440 ts_tablet_manager.cc:1397] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:33.580461 1440 raft_consensus.cc:348] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35913 } } peers { permanent_uuid: "cfeba1a35d2948dba0fc684225eb0903" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37339 } } peers { permanent_uuid: "71ba7aec4edd4515a5fa91755d6201b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41899 } }
I20260620 09:16:33.580590 1440 raft_consensus.cc:374] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:33.580610 1440 raft_consensus.cc:729] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb350fb30944cbdbd14c9f93898b7cb, State: Initialized, Role: FOLLOWER
I20260620 09:16:33.580681 1440 consensus_queue.cc:260] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35913 } } peers { permanent_uuid: "cfeba1a35d2948dba0fc684225eb0903" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37339 } } peers { permanent_uuid: "71ba7aec4edd4515a5fa91755d6201b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41899 } }
I20260620 09:16:33.580869 1440 ts_tablet_manager.cc:1428] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:16:33.580909 1287 heartbeater.cc:503] Master 127.0.98.62:46865 was elected leader, sending a full tablet report...
W20260620 09:16:33.645569 1288 tablet.cc:2369] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:16:33.655077 1443 raft_consensus.cc:482] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:33.655189 1443 raft_consensus.cc:504] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35913 } } peers { permanent_uuid: "cfeba1a35d2948dba0fc684225eb0903" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37339 } } peers { permanent_uuid: "71ba7aec4edd4515a5fa91755d6201b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41899 } }
I20260620 09:16:33.655576 1443 leader_election.cc:288] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfeba1a35d2948dba0fc684225eb0903 (127.0.98.1:37339), 71ba7aec4edd4515a5fa91755d6201b2 (127.0.98.3:41899)
I20260620 09:16:33.658399 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.655809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51511 (local address 127.0.98.1:37339)
0620 09:16:33.655853 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.655855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.655864 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:16:33.655928 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.655929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.655969 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.656019 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:33.656196 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.656676 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.657716 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.658099 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.658162 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.658173 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.658299 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.658336 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:16:33.658399 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.655747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37339 (local address 127.0.98.2:51511)
0620 09:16:33.655803 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:33.655815 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:33.655872 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:33.656064 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:33.656065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:33.656067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:33.656128 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.656130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.656778 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.656780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.657621 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.657624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.658170 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.658170 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.658241 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.658244 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:33.658288 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 09:16:33.658326 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":17}
I20260620 09:16:33.658759 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d474f5cc2444c9bb9b553cb1d9c7dd6" candidate_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfeba1a35d2948dba0fc684225eb0903" is_pre_election: true
I20260620 09:16:33.658941 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.655840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41899 (local address 127.0.98.2:43797)
0620 09:16:33.655992 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:33.656002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:33.656064 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:33.656764 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:33.656765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:33.656768 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:33.656840 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.656842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.657425 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.657426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.658138 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:33.658140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.658727 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.658728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:33.658791 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.658794 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:33.658834 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 09:16:33.658862 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260620 09:16:33.658946 1107 raft_consensus.cc:2454] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eb350fb30944cbdbd14c9f93898b7cb in term 0.
I20260620 09:16:33.658969 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.655968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43797 (local address 127.0.98.3:41899)
0620 09:16:33.656023 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.656025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.656051 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:16:33.656120 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.656121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.656131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.656188 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:33.656910 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.657353 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.658221 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.658654 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.658721 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.658730 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.658855 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.658907 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:16:33.659157 1175 leader_election.cc:302] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb350fb30944cbdbd14c9f93898b7cb, cfeba1a35d2948dba0fc684225eb0903; no voters:
I20260620 09:16:33.659265 1443 raft_consensus.cc:2781] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:33.659319 1443 raft_consensus.cc:482] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:33.659339 1443 raft_consensus.cc:3037] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:33.659307 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d474f5cc2444c9bb9b553cb1d9c7dd6" candidate_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ba7aec4edd4515a5fa91755d6201b2" is_pre_election: true
I20260620 09:16:33.659448 1375 raft_consensus.cc:2454] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eb350fb30944cbdbd14c9f93898b7cb in term 0.
I20260620 09:16:33.660132 1443 raft_consensus.cc:504] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35913 } } peers { permanent_uuid: "cfeba1a35d2948dba0fc684225eb0903" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37339 } } peers { permanent_uuid: "71ba7aec4edd4515a5fa91755d6201b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41899 } }
I20260620 09:16:33.660247 1443 leader_election.cc:288] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [CANDIDATE]: Term 1 election: Requested vote from peers cfeba1a35d2948dba0fc684225eb0903 (127.0.98.1:37339), 71ba7aec4edd4515a5fa91755d6201b2 (127.0.98.3:41899)
I20260620 09:16:33.660463 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d474f5cc2444c9bb9b553cb1d9c7dd6" candidate_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ba7aec4edd4515a5fa91755d6201b2"
I20260620 09:16:33.660461 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d474f5cc2444c9bb9b553cb1d9c7dd6" candidate_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfeba1a35d2948dba0fc684225eb0903"
I20260620 09:16:33.660534 1375 raft_consensus.cc:3037] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:33.660537 1107 raft_consensus.cc:3037] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:33.661194 1375 raft_consensus.cc:2454] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 71ba7aec4edd4515a5fa91755d6201b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eb350fb30944cbdbd14c9f93898b7cb in term 1.
I20260620 09:16:33.661316 1107 raft_consensus.cc:2454] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P cfeba1a35d2948dba0fc684225eb0903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eb350fb30944cbdbd14c9f93898b7cb in term 1.
I20260620 09:16:33.661339 1175 leader_election.cc:302] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb350fb30944cbdbd14c9f93898b7cb, 71ba7aec4edd4515a5fa91755d6201b2; no voters:
I20260620 09:16:33.661433 1443 raft_consensus.cc:2781] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:33.661589 1443 raft_consensus.cc:686] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [term 1 LEADER]: Becoming Leader. State: Replica: 2eb350fb30944cbdbd14c9f93898b7cb, State: Running, Role: LEADER
I20260620 09:16:33.661675 1443 consensus_queue.cc:237] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35913 } } peers { permanent_uuid: "cfeba1a35d2948dba0fc684225eb0903" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37339 } } peers { permanent_uuid: "71ba7aec4edd4515a5fa91755d6201b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41899 } }
I20260620 09:16:33.662494 962 catalog_manager.cc:5495] T 1d474f5cc2444c9bb9b553cb1d9c7dd6 P 2eb350fb30944cbdbd14c9f93898b7cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 2eb350fb30944cbdbd14c9f93898b7cb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb350fb30944cbdbd14c9f93898b7cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfeba1a35d2948dba0fc684225eb0903" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ba7aec4edd4515a5fa91755d6201b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41899 } health_report { overall_health: UNKNOWN } } }
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:16:33.686617 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.679919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.0.98.1:37339)
0620 09:16:33.679971 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.679974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.680081 (+ 107us) server_negotiation.cc:407] Connection header received
0620 09:16:33.680152 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.680152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.680165 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.680211 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:33.680732 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.681203 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.681885 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.682070 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.682124 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.684269 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:33.684280 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:33.684280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:33.684287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:33.685597 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.685903 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.685907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.685907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.685960 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.686120 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.686121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.686121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.686262 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:33.686412 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.686446 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.686523 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:16:33.707165 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.700682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.0.98.1:37339)
0620 09:16:33.700754 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.700756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.700816 (+ 60us) server_negotiation.cc:407] Connection header received
0620 09:16:33.700898 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.700899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.700909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.700951 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:33.701463 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.701937 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.702636 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.702842 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.702897 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.704949 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:33.704950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:33.704950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:33.704957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:33.706141 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.706448 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.706449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.706450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.706499 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.706656 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.706657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.706657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.706749 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:33.706842 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.706996 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.707072 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:16:33.727874 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:33.721306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.0.98.1:37339)
0620 09:16:33.721361 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:33.721363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:33.721485 (+ 122us) server_negotiation.cc:407] Connection header received
0620 09:16:33.721557 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:33.721557 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:33.721567 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:33.721616 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:33.722111 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.722610 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.723299 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:33.723484 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:33.723539 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:33.725714 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:33.725715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:33.725716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:33.725723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:33.726944 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.727234 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.727235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.727236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.727299 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:33.727425 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:33.727425 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:33.727425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:33.727515 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:33.727588 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:33.727718 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 09:16:33.727783 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 09:16:33.728322 1067 tablet_service.cc:2908] Rejecting scan request for tablet 1d474f5cc2444c9bb9b553cb1d9c7dd6: Uninitialized: clean time has not yet been initialized
I20260620 09:16:33.729092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 1024
I20260620 09:16:33.733567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 1158
I20260620 09:16:33.737848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 1292
I20260620 09:16:33.741492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 930
2026-06-20T09:16:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1844 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[1459](info): set up 2 sockets
Jun 20 09:16:33 dist-test-slave-rq8c krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260620 09:16:34.694962 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:16:34 dist-test-slave-rq8c krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:16:34.800029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44121
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:34.878230 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:34.878427 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:34.878448 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:34.879552 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:34.879585 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 09:16:34.879597 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:34.879653 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:34.879671 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:34.879689 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:34.881181 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44121
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:34.881376 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:34.881536 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:34.886286 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:34.886363 1472 server_base.cc:1034] running on GCE node
W20260620 09:16:34.886307 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:34.886286 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:34.886693 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:34.886958 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:34.888130 1472 hybrid_clock.cc:648] HybridClock initialized: now 1781946994888115 us; error 29 us; skew 500 ppm
Jun 20 09:16:34 dist-test-slave-rq8c krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:34.890573 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:16:34.891047 1472 webserver.cc:460] Webserver started at http://127.0.98.62:46659/ using document root <none> and password file <none>
I20260620 09:16:34.891254 1472 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:34.891305 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:34.891429 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:34.892494 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "445fbe4e95394a97b5c192d5a891d97f"
format_stamp: "Formatted at 2026-06-20 09:16:34 on dist-test-slave-rq8c"
server_key: "243472f0ae37530ef6c225fbe9d2cc33"
server_key_iv: "fe9d430a72a1de32dfc4d1fc9fdb9ae1"
server_key_version: "encryptionkey@0"
I20260620 09:16:34.892836 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "445fbe4e95394a97b5c192d5a891d97f"
format_stamp: "Formatted at 2026-06-20 09:16:34 on dist-test-slave-rq8c"
server_key: "243472f0ae37530ef6c225fbe9d2cc33"
server_key_iv: "fe9d430a72a1de32dfc4d1fc9fdb9ae1"
server_key_version: "encryptionkey@0"
I20260620 09:16:34.894142 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:34.894888 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:34.895030 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:34.895113 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "445fbe4e95394a97b5c192d5a891d97f"
format_stamp: "Formatted at 2026-06-20 09:16:34 on dist-test-slave-rq8c"
server_key: "243472f0ae37530ef6c225fbe9d2cc33"
server_key_iv: "fe9d430a72a1de32dfc4d1fc9fdb9ae1"
server_key_version: "encryptionkey@0"
I20260620 09:16:34.895179 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:34.909283 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:34.911309 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:34.911440 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:34.915747 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44121
I20260620 09:16:34.915799 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44121 every 8 connection(s)
I20260620 09:16:34.916085 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:34.916673 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:34.918746 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f: Bootstrap starting.
I20260620 09:16:34.919269 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:34.919486 1550 log.cc:824] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:34.920053 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f: No bootstrap required, opened a new log
I20260620 09:16:34.921901 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445fbe4e95394a97b5c192d5a891d97f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44121 } }
I20260620 09:16:34.922030 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:34.922051 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445fbe4e95394a97b5c192d5a891d97f, State: Initialized, Role: FOLLOWER
I20260620 09:16:34.922154 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445fbe4e95394a97b5c192d5a891d97f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44121 } }
I20260620 09:16:34.922235 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:34.922264 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:34.922293 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:34.922724 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445fbe4e95394a97b5c192d5a891d97f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44121 } }
I20260620 09:16:34.922801 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 445fbe4e95394a97b5c192d5a891d97f; no voters:
I20260620 09:16:34.922917 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:34.922967 1553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:34.923116 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:34.923141 1553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [term 1 LEADER]: Becoming Leader. State: Replica: 445fbe4e95394a97b5c192d5a891d97f, State: Running, Role: LEADER
I20260620 09:16:34.923290 1553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445fbe4e95394a97b5c192d5a891d97f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44121 } }
I20260620 09:16:34.923599 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 445fbe4e95394a97b5c192d5a891d97f. Latest consensus state: current_term: 1 leader_uuid: "445fbe4e95394a97b5c192d5a891d97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445fbe4e95394a97b5c192d5a891d97f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44121 } } }
I20260620 09:16:34.923646 1554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "445fbe4e95394a97b5c192d5a891d97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "445fbe4e95394a97b5c192d5a891d97f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44121 } } }
I20260620 09:16:34.923722 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:34.923755 1554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:34.923897 1562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:34.924371 1562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:34.924450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 1472
I20260620 09:16:34.924542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:34.924682 392 external_mini_cluster.cc:1428] Setting key 0e1e58da841d7924dce80fd1c3f8e619
I20260620 09:16:34.925784 1562 catalog_manager.cc:1347] Generated new cluster ID: c7789481649d416bb375536c2f4578be
I20260620 09:16:34.925830 1562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:16:34 dist-test-slave-rq8c krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:34.935514 1562 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:34.935835 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:34.925229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.0.98.62:44121)
0620 09:16:34.925604 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:34.925612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:34.925641 (+ 29us) server_negotiation.cc:407] Connection header received
0620 09:16:34.925748 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:34.925755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:34.925807 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:34.925932 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:34.926564 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:34.928015 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:34.929005 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:34.929652 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:34.929758 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:34.932337 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:34.932341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:34.932341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:34.932353 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:34.934179 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:34.934563 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:34.934565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:34.934566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:34.934665 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:34.934854 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:34.934855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:34.934855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:34.935103 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:34.935262 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:34.935350 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 09:16:34.935573 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":66,"threads_started":1}
I20260620 09:16:34.936167 1562 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:34.940901 1562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f: Generated new TSK 0
I20260620 09:16:34.941072 1562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:16:34.967499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44121
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:35.046101 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:35.046303 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:35.046324 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:35.047435 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:35.047467 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:35.047487 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:16:35.047508 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:35.048980 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:35.049175 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:35.049348 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:35.053958 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:35.054147 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:35.054067 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:35.054231 1576 server_base.cc:1034] running on GCE node
I20260620 09:16:35.054399 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:35.054608 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:35.055738 1576 hybrid_clock.cc:648] HybridClock initialized: now 1781946995055722 us; error 30 us; skew 500 ppm
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:35.058094 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:16:35.058466 1576 webserver.cc:460] Webserver started at http://127.0.98.1:40823/ using document root <none> and password file <none>
I20260620 09:16:35.058645 1576 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:35.058682 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:35.058772 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:35.059823 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "546536e7b2e24a46add30644ae2587b4"
format_stamp: "Formatted at 2026-06-20 09:16:35 on dist-test-slave-rq8c"
server_key: "fa1c8461a399eb7cfe0a0bd6f7d46492"
server_key_iv: "4373550b6b3813166ebd1be5879a1e56"
server_key_version: "encryptionkey@0"
I20260620 09:16:35.060119 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "546536e7b2e24a46add30644ae2587b4"
format_stamp: "Formatted at 2026-06-20 09:16:35 on dist-test-slave-rq8c"
server_key: "fa1c8461a399eb7cfe0a0bd6f7d46492"
server_key_iv: "4373550b6b3813166ebd1be5879a1e56"
server_key_version: "encryptionkey@0"
I20260620 09:16:35.061322 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:35.062000 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:35.062198 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:35.062278 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "546536e7b2e24a46add30644ae2587b4"
format_stamp: "Formatted at 2026-06-20 09:16:35 on dist-test-slave-rq8c"
server_key: "fa1c8461a399eb7cfe0a0bd6f7d46492"
server_key_iv: "4373550b6b3813166ebd1be5879a1e56"
server_key_version: "encryptionkey@0"
I20260620 09:16:35.062335 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:35.078307 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:35.080497 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:35.080621 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:35.081089 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:35.081144 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:35.081167 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:35.081180 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:35.087080 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40489
I20260620 09:16:35.087219 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40489 every 8 connection(s)
I20260620 09:16:35.087517 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:35.090517 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.081551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53461 (local address 127.0.98.62:44121)
0620 09:16:35.081628 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.081630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.081790 (+ 160us) server_negotiation.cc:407] Connection header received
0620 09:16:35.082332 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.082334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.082346 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.082426 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:35.083226 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.084005 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.085443 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.085755 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.085825 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.088270 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:35.088272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:35.088273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:35.088278 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:35.089367 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.089738 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.089739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.089740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.089799 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.090003 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.090004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.090005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.090128 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:35.090188 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.090388 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.090454 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:16:35.090682 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.081507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44121 (local address 127.0.98.1:53461)
0620 09:16:35.081661 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.081702 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.082230 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.082492 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.082497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.082838 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:35.083123 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.083128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.084091 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.084091 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.085347 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.085350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.085830 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.085830 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.085950 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.086499 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:35.086509 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:35.088191 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:35.089435 (+ 1244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.089441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.089443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.089653 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.089854 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.089855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.089855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.089940 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.090181 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:35.090184 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:35.090292 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.090367 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.090479 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260620 09:16:35.091220 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44121
I20260620 09:16:35.091303 1707 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:35.091475 1707 heartbeater.cc:511] Master 127.0.98.62:44121 requested a full tablet report, sending...
I20260620 09:16:35.091990 1504 ts_manager.cc:194] Registered new tserver with Master: 546536e7b2e24a46add30644ae2587b4 (127.0.98.1:40489)
I20260620 09:16:35.092839 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53461
I20260620 09:16:35.094795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 1576
I20260620 09:16:35.094888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:16:35.095057 392 external_mini_cluster.cc:1428] Setting key d036ae4b89b3c156d42021fcddfe4eb8
I20260620 09:16:35.100924 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.095393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44431 (local address 127.0.98.62:44121)
0620 09:16:35.095427 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.095429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.095764 (+ 335us) server_negotiation.cc:407] Connection header received
0620 09:16:35.095829 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.095830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.095841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.095877 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:35.096208 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.096760 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.097568 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.097807 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.097839 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.098747 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:35.098748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:35.098749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:35.098755 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:35.099992 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.100274 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.100275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.100275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.100323 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.100477 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.100477 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.100478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.100575 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:35.100630 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.100762 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.100836 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:16:35.100934 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.095350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44121 (local address 127.0.98.1:44431)
0620 09:16:35.095707 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.095719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.095776 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.095891 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.095892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.096078 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:35.096147 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.096149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.096844 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.096845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.097475 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.097478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.097898 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.097898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.097944 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.098344 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:35.098348 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:35.098647 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:35.100059 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.100060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.100060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.100215 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.100373 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.100373 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.100374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.100429 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.100634 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:35.100634 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:35.100676 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.100735 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.100827 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:16:35.117796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44121
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:35.196249 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:35.196448 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:35.196480 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:35.197662 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:35.197705 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:35.197732 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:16:35.197757 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:35.199369 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:35.199646 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:35.199862 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:35.204869 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:35.204864 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:35.204864 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:35.205029 1716 server_base.cc:1034] running on GCE node
I20260620 09:16:35.205243 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:35.205477 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:35.206627 1716 hybrid_clock.cc:648] HybridClock initialized: now 1781946995206602 us; error 38 us; skew 500 ppm
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:35.209270 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:16:35.209726 1716 webserver.cc:460] Webserver started at http://127.0.98.2:44029/ using document root <none> and password file <none>
I20260620 09:16:35.209946 1716 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:35.209992 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:35.210103 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:35.211165 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3"
format_stamp: "Formatted at 2026-06-20 09:16:35 on dist-test-slave-rq8c"
server_key: "c1dbb5513bcde7a26c2402fbdc6c80ae"
server_key_iv: "339b8fae54e9478f12d8c3100ff9ab83"
server_key_version: "encryptionkey@0"
I20260620 09:16:35.211508 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3"
format_stamp: "Formatted at 2026-06-20 09:16:35 on dist-test-slave-rq8c"
server_key: "c1dbb5513bcde7a26c2402fbdc6c80ae"
server_key_iv: "339b8fae54e9478f12d8c3100ff9ab83"
server_key_version: "encryptionkey@0"
I20260620 09:16:35.212986 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:35.213763 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:35.213953 1716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:35.214040 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3"
format_stamp: "Formatted at 2026-06-20 09:16:35 on dist-test-slave-rq8c"
server_key: "c1dbb5513bcde7a26c2402fbdc6c80ae"
server_key_iv: "339b8fae54e9478f12d8c3100ff9ab83"
server_key_version: "encryptionkey@0"
I20260620 09:16:35.214098 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:35.229377 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:35.231562 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:35.231748 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:35.232282 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:35.232331 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:35.232352 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:35.232365 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:35.238303 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43339
I20260620 09:16:35.238409 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43339 every 8 connection(s)
I20260620 09:16:35.238679 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:35.242014 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.232748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53583 (local address 127.0.98.62:44121)
0620 09:16:35.232815 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.232817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.233041 (+ 224us) server_negotiation.cc:407] Connection header received
0620 09:16:35.233578 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.233579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.233591 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.233629 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:35.234475 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.235237 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.236770 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.237061 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.237137 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.239586 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:35.239588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:35.239588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:35.239595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:35.240924 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.241288 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.241289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.241290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.241345 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.241527 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.241527 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.241527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.241625 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:35.241670 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.241874 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.241932 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:16:35.242156 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.232717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44121 (local address 127.0.98.2:53583)
0620 09:16:35.232941 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.232971 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.233479 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.233717 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.233721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.234063 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:35.234360 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.234367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.235335 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.235336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.236675 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.236677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.237163 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.237164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.237316 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.237911 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:35.237919 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:35.239504 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:35.241000 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.241005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.241007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.241218 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.241397 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.241398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.241398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.241486 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.241665 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:35.241668 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:35.241779 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.241853 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.241971 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":64,"threads_started":1}
I20260620 09:16:35.242184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 1716
I20260620 09:16:35.242274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:16:35.242445 392 external_mini_cluster.cc:1428] Setting key ebf19f7b11e7cd88460e28d1f646aa84
I20260620 09:16:35.242791 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44121
I20260620 09:16:35.242875 1847 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:35.243046 1847 heartbeater.cc:511] Master 127.0.98.62:44121 requested a full tablet report, sending...
I20260620 09:16:35.243396 1503 ts_manager.cc:194] Registered new tserver with Master: ce35eae8f9ae41bb98084d25ebfcd4f3 (127.0.98.2:43339)
I20260620 09:16:35.243867 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53583
I20260620 09:16:35.248934 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.243536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51755 (local address 127.0.98.62:44121)
0620 09:16:35.243570 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.243572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.243690 (+ 118us) server_negotiation.cc:407] Connection header received
0620 09:16:35.243751 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.243752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.243760 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.243811 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:35.244089 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.244601 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.245555 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.245753 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.245802 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.246689 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:35.246690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:35.246691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:35.246698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:35.247921 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.248217 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.248218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.248219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.248269 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.248422 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.248423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.248423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.248566 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:35.248622 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.248767 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.248840 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:16:35.248945 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.243499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44121 (local address 127.0.98.2:51755)
0620 09:16:35.243643 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.243653 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.243702 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.243816 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.243817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.243973 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:35.244037 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.244039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.244704 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.244705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.245477 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.245479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.245808 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.245809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.245866 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.246285 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:35.246289 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:35.246615 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:35.247990 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.247990 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.247991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.248159 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.248320 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.248320 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.248321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.248375 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.248623 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:35.248624 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:35.248683 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.248739 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.248841 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:16:35.264256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44121
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:35.342549 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:35.342761 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:35.342793 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:35.343999 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:35.344045 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:35.344072 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:16:35.344094 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:35.345661 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:35.345891 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:35.346089 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:35.350718 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:35.350760 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:35.350811 1856 server_base.cc:1034] running on GCE node
W20260620 09:16:35.350719 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:35.351163 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:35.351390 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:35.352524 1856 hybrid_clock.cc:648] HybridClock initialized: now 1781946995352514 us; error 30 us; skew 500 ppm
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:35.354773 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:16:35.355176 1856 webserver.cc:460] Webserver started at http://127.0.98.3:45399/ using document root <none> and password file <none>
I20260620 09:16:35.355363 1856 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:35.355407 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:35.355527 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:35.356566 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "27bc63eea961489c8f121fc32fb3184b"
format_stamp: "Formatted at 2026-06-20 09:16:35 on dist-test-slave-rq8c"
server_key: "3939b5b5a65b469afb34432988fd6b8d"
server_key_iv: "74b2de5937a4130e10a6ad021b4a0130"
server_key_version: "encryptionkey@0"
I20260620 09:16:35.356884 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "27bc63eea961489c8f121fc32fb3184b"
format_stamp: "Formatted at 2026-06-20 09:16:35 on dist-test-slave-rq8c"
server_key: "3939b5b5a65b469afb34432988fd6b8d"
server_key_iv: "74b2de5937a4130e10a6ad021b4a0130"
server_key_version: "encryptionkey@0"
I20260620 09:16:35.358071 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:16:35.358680 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:35.358827 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:35.358904 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "27bc63eea961489c8f121fc32fb3184b"
format_stamp: "Formatted at 2026-06-20 09:16:35 on dist-test-slave-rq8c"
server_key: "3939b5b5a65b469afb34432988fd6b8d"
server_key_iv: "74b2de5937a4130e10a6ad021b4a0130"
server_key_version: "encryptionkey@0"
I20260620 09:16:35.358963 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:35.368871 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:35.370818 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:35.370951 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:35.371402 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:35.371439 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:35.371469 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:35.371485 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:35.377475 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43055
I20260620 09:16:35.377550 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43055 every 8 connection(s)
I20260620 09:16:35.377816 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:16:35.378216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 1856
I20260620 09:16:35.378310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:16:35.378474 392 external_mini_cluster.cc:1428] Setting key 13139f9f8c716cb0d11e6903a2d741a7
I20260620 09:16:35.380993 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.371808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59387 (local address 127.0.98.62:44121)
0620 09:16:35.371874 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.371877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.372019 (+ 142us) server_negotiation.cc:407] Connection header received
0620 09:16:35.372488 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.372489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.372502 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.372578 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:35.373294 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.374046 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.375442 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.375735 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.375793 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.377809 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:35.377811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:35.377812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:35.377820 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:35.379738 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.380095 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.380096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.380096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.380159 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.380377 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.380378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.380379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.380508 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:35.380578 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.380856 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.380930 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:16:35.381161 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.371773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44121 (local address 127.0.98.3:59387)
0620 09:16:35.371899 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.371936 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.372397 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.372655 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.372659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.372925 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:35.373193 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.373198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.374136 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.374137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.375368 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.375371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.375805 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.375806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.375930 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.376398 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:35.376405 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:35.377719 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:35.379804 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.379810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.379811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.380032 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.380205 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.380205 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.380206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.380301 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.380596 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:35.380601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:35.380738 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.380834 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.380961 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":67,"threads_started":1}
I20260620 09:16:35.381621 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44121
I20260620 09:16:35.381705 1987 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:35.381888 1987 heartbeater.cc:511] Master 127.0.98.62:44121 requested a full tablet report, sending...
I20260620 09:16:35.382277 1503 ts_manager.cc:194] Registered new tserver with Master: 27bc63eea961489c8f121fc32fb3184b (127.0.98.3:43055)
I20260620 09:16:35.382740 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59387
I20260620 09:16:35.387490 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.382671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52081 (local address 127.0.98.62:44121)
0620 09:16:35.382857 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.382859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.382867 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:35.382884 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.382884 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.382893 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.382932 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:35.383281 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.383822 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.384480 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.384674 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.384731 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.385533 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:35.385534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:35.385535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:35.385541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:35.386562 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.386843 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.386844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.386845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.386894 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.387049 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.387050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.387050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.387152 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:35.387205 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.387337 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.387401 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260620 09:16:35.387503 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.382609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44121 (local address 127.0.98.3:52081)
0620 09:16:35.382729 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.382744 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.382792 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.382949 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.382949 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.383123 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:35.383211 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.383213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.383899 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.383900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.384412 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.384414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.384734 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.384735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.384796 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.385203 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:35.385207 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:35.385458 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:35.386629 (+ 1171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.386630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.386630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.386784 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.386943 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.386944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.386944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.386999 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.387206 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:35.387207 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:35.387249 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.387309 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.387400 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260620 09:16:35.390019 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:35.418298 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.411264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.0.98.62:44121)
0620 09:16:35.411334 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.411336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.411347 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:16:35.411392 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.411393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.411404 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.411485 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:35.412436 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.413059 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.413794 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.413991 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.414045 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.416227 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:35.416228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:35.416229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:35.416237 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:35.417348 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.417634 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.417635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.417636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.417686 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.417849 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.417849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.417850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.417953 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:35.418036 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.418135 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.418208 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:16:35.419768 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:35.421288 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:35.427390 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.423319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.0.98.1:40489)
0620 09:16:35.423798 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.423801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.423811 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:16:35.423940 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.423946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.423965 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.424223 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:35.424282 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.425353 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.426530 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.427014 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.427077 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.427157 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.427255 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.427291 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":75,"threads_started":1}
I20260620 09:16:35.427412 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.423422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43339 (local address 127.0.0.1:36516)
0620 09:16:35.423562 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.423570 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.423607 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.423972 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.423975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.423977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:35.424050 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.424052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.424993 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.424994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.426226 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.426229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.427203 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.427204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.427254 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.427257 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.427297 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.427327 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":33,"threads_started":1}
I20260620 09:16:35.427749 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.423525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.0.98.3:43055)
0620 09:16:35.423796 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.423801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.423811 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:16:35.423848 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.423850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.423865 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.424181 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:35.424242 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.425350 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.426400 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.427081 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.427462 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.427546 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.427567 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.427602 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":115,"threads_started":1}
I20260620 09:16:35.428021 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.423523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.0.98.2:43339)
0620 09:16:35.423738 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.423742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.423751 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:16:35.423806 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.423808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.423823 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.423893 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:35.424141 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.424931 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.426307 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.427019 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.427086 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.427177 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.427549 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.427594 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":106,"threads_started":1}
I20260620 09:16:35.427425 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.423238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40489 (local address 127.0.0.1:57798)
0620 09:16:35.423487 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.423497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.423545 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.424099 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.424100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.424101 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:35.424156 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.424157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.425429 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.425430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.426464 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.426466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.427087 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.427088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.427169 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.427182 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.427244 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.427297 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":30,"thread_start_us":51,"threads_started":1}
I20260620 09:16:35.427426 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.423217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43055 (local address 127.0.0.1:38500)
0620 09:16:35.423331 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.423370 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.423430 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.424045 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.424046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.424048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:35.424110 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.424111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.425451 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.425452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.426313 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.426315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.427164 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.427166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.427233 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.427237 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.427265 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.427308 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260620 09:16:35.428804 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 40f58e92bde0470f90e387c94d487076 (DEFAULT_TABLE table=test-table [id=934d096673cd41bdb7983249e27d2499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:35.428804 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 40f58e92bde0470f90e387c94d487076 (DEFAULT_TABLE table=test-table [id=934d096673cd41bdb7983249e27d2499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:35.429028 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 40f58e92bde0470f90e387c94d487076 (DEFAULT_TABLE table=test-table [id=934d096673cd41bdb7983249e27d2499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:35.429168 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f58e92bde0470f90e387c94d487076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:35.429168 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f58e92bde0470f90e387c94d487076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:35.429342 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f58e92bde0470f90e387c94d487076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:35.432155 2007 tablet_bootstrap.cc:492] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4: Bootstrap starting.
I20260620 09:16:35.433049 2007 tablet_bootstrap.cc:654] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:35.433102 2008 tablet_bootstrap.cc:492] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b: Bootstrap starting.
I20260620 09:16:35.433398 2007 log.cc:824] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:35.433679 2008 tablet_bootstrap.cc:654] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:35.433743 2009 tablet_bootstrap.cc:492] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3: Bootstrap starting.
I20260620 09:16:35.433979 2008 log.cc:824] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:35.434095 2007 tablet_bootstrap.cc:492] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4: No bootstrap required, opened a new log
I20260620 09:16:35.434166 2007 ts_tablet_manager.cc:1397] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:35.434319 2009 tablet_bootstrap.cc:654] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:35.434630 2009 log.cc:824] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:35.434654 2008 tablet_bootstrap.cc:492] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b: No bootstrap required, opened a new log
I20260620 09:16:35.434713 2008 ts_tablet_manager.cc:1397] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:35.435302 2009 tablet_bootstrap.cc:492] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3: No bootstrap required, opened a new log
I20260620 09:16:35.435366 2009 ts_tablet_manager.cc:1397] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:35.435858 2007 raft_consensus.cc:348] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.435986 2007 raft_consensus.cc:374] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:35.436017 2007 raft_consensus.cc:729] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546536e7b2e24a46add30644ae2587b4, State: Initialized, Role: FOLLOWER
I20260620 09:16:35.436129 2007 consensus_queue.cc:260] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.436414 2007 ts_tablet_manager.cc:1428] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:16:35.436524 1707 heartbeater.cc:503] Master 127.0.98.62:44121 was elected leader, sending a full tablet report...
I20260620 09:16:35.436551 2008 raft_consensus.cc:348] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.436702 2008 raft_consensus.cc:374] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:35.436735 2008 raft_consensus.cc:729] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27bc63eea961489c8f121fc32fb3184b, State: Initialized, Role: FOLLOWER
I20260620 09:16:35.436851 2008 consensus_queue.cc:260] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.437098 2008 ts_tablet_manager.cc:1428] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:16:35.437121 1987 heartbeater.cc:503] Master 127.0.98.62:44121 was elected leader, sending a full tablet report...
I20260620 09:16:35.437832 2009 raft_consensus.cc:348] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.437978 2009 raft_consensus.cc:374] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:35.438002 2009 raft_consensus.cc:729] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce35eae8f9ae41bb98084d25ebfcd4f3, State: Initialized, Role: FOLLOWER
I20260620 09:16:35.438118 2009 consensus_queue.cc:260] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.438330 2009 ts_tablet_manager.cc:1428] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:16:35.438369 1847 heartbeater.cc:503] Master 127.0.98.62:44121 was elected leader, sending a full tablet report...
I20260620 09:16:35.445417 2015 raft_consensus.cc:482] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:35.445511 2015 raft_consensus.cc:504] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.445837 2015 leader_election.cc:288] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 546536e7b2e24a46add30644ae2587b4 (127.0.98.1:40489), 27bc63eea961489c8f121fc32fb3184b (127.0.98.3:43055)
I20260620 09:16:35.448498 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.446160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47157 (local address 127.0.98.3:43055)
0620 09:16:35.446238 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.446239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.446246 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:35.446273 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.446273 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.446282 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.446325 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:35.446453 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.446920 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.447854 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.448200 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.448258 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.448270 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.448398 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.448433 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:16:35.448509 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.446010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43055 (local address 127.0.98.2:47157)
0620 09:16:35.446066 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.446076 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.446112 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.446338 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.446339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.446340 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:35.446402 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.446403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.446988 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.446989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.447780 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.447782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.448263 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.448264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.448343 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.448346 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.448389 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.448428 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 09:16:35.448551 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.446098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59443 (local address 127.0.98.1:40489)
0620 09:16:35.446139 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.446142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.446294 (+ 152us) server_negotiation.cc:407] Connection header received
0620 09:16:35.446351 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.446352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.446362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.446413 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:35.446539 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.446997 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.447912 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.448268 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.448326 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.448334 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.448459 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.448492 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:16:35.448832 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f58e92bde0470f90e387c94d487076" candidate_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bc63eea961489c8f121fc32fb3184b" is_pre_election: true
I20260620 09:16:35.448891 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.446012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40489 (local address 127.0.98.2:59443)
0620 09:16:35.446227 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.446237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.446293 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.446417 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.446417 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.446419 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:35.446487 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.446488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.447063 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.447064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.447841 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.447842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.448333 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.448334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.448399 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.448406 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.448449 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.448486 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":68,"thread_start_us":102,"threads_started":1}
I20260620 09:16:35.448969 1941 raft_consensus.cc:2454] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce35eae8f9ae41bb98084d25ebfcd4f3 in term 0.
I20260620 09:16:35.449182 1735 leader_election.cc:302] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27bc63eea961489c8f121fc32fb3184b, ce35eae8f9ae41bb98084d25ebfcd4f3; no voters:
I20260620 09:16:35.449184 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f58e92bde0470f90e387c94d487076" candidate_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546536e7b2e24a46add30644ae2587b4" is_pre_election: true
I20260620 09:16:35.449311 1661 raft_consensus.cc:2454] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce35eae8f9ae41bb98084d25ebfcd4f3 in term 0.
I20260620 09:16:35.449366 2015 raft_consensus.cc:2781] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:35.449429 2015 raft_consensus.cc:482] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:35.449457 2015 raft_consensus.cc:3037] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:35.450364 2015 raft_consensus.cc:504] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.450485 2015 leader_election.cc:288] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [CANDIDATE]: Term 1 election: Requested vote from peers 546536e7b2e24a46add30644ae2587b4 (127.0.98.1:40489), 27bc63eea961489c8f121fc32fb3184b (127.0.98.3:43055)
I20260620 09:16:35.450624 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f58e92bde0470f90e387c94d487076" candidate_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546536e7b2e24a46add30644ae2587b4"
I20260620 09:16:35.450703 1661 raft_consensus.cc:3037] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:35.451391 1661 raft_consensus.cc:2454] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce35eae8f9ae41bb98084d25ebfcd4f3 in term 1.
I20260620 09:16:35.451489 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f58e92bde0470f90e387c94d487076" candidate_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bc63eea961489c8f121fc32fb3184b"
I20260620 09:16:35.451558 1734 leader_election.cc:302] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546536e7b2e24a46add30644ae2587b4, ce35eae8f9ae41bb98084d25ebfcd4f3; no voters:
I20260620 09:16:35.451565 1941 raft_consensus.cc:3037] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:35.451663 2015 raft_consensus.cc:2781] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:35.451807 2015 raft_consensus.cc:686] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 1 LEADER]: Becoming Leader. State: Replica: ce35eae8f9ae41bb98084d25ebfcd4f3, State: Running, Role: LEADER
I20260620 09:16:35.451889 2015 consensus_queue.cc:237] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.452414 1941 raft_consensus.cc:2454] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce35eae8f9ae41bb98084d25ebfcd4f3 in term 1.
I20260620 09:16:35.452737 1504 catalog_manager.cc:5495] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce35eae8f9ae41bb98084d25ebfcd4f3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } health_report { overall_health: UNKNOWN } } }
I20260620 09:16:35.465528 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.463083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.0.98.2:43339)
0620 09:16:35.463161 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.463163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.463171 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:35.463204 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.463205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.463222 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.463275 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:35.463403 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.463921 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.464723 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.464943 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.464992 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.465190 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:35.465309 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:35.465376 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.465425 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.465446 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:16:35.467347 1941 raft_consensus.cc:1261] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b [term 1 FOLLOWER]: Refusing update from remote peer ce35eae8f9ae41bb98084d25ebfcd4f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:35.467347 1661 raft_consensus.cc:1261] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4 [term 1 FOLLOWER]: Refusing update from remote peer ce35eae8f9ae41bb98084d25ebfcd4f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:35.467737 2017 consensus_queue.cc:1035] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:35.467870 2015 consensus_queue.cc:1035] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:35.470062 2020 mvcc.cc:205] Tried to move back new op lower bound from 7298854893431611392 to 7298854893371662336. Current Snapshot: MvccSnapshot[applied={T|T < 7298854893431611392}]
I20260620 09:16:35.470244 2022 mvcc.cc:205] Tried to move back new op lower bound from 7298854893431611392 to 7298854893371662336. Current Snapshot: MvccSnapshot[applied={T|T < 7298854893431611392}]
I20260620 09:16:35.471841 2021 mvcc.cc:205] Tried to move back new op lower bound from 7298854893431611392 to 7298854893371662336. Current Snapshot: MvccSnapshot[applied={T|T < 7298854893431611392 or (T in {7298854893431611392})}]
I20260620 09:16:35.473428 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:35.473534 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:35.474339 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f: Sending DeleteTablet for 3 replicas of tablet 40f58e92bde0470f90e387c94d487076
I20260620 09:16:35.474761 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 40f58e92bde0470f90e387c94d487076 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57798
I20260620 09:16:35.474761 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 40f58e92bde0470f90e387c94d487076 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38500
I20260620 09:16:35.475049 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 40f58e92bde0470f90e387c94d487076 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36516
I20260620 09:16:35.475296 2032 tablet_replica.cc:330] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3: stopping tablet replica
I20260620 09:16:35.475296 2030 tablet_replica.cc:330] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b: stopping tablet replica
I20260620 09:16:35.475430 2032 raft_consensus.cc:2229] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:35.475437 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 09:16:35.475543 2032 raft_consensus.cc:2258] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 09:16:35.475587 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:35.476063 2032 ts_tablet_manager.cc:1905] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:35.477067 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5bedbd8cba4140b866014052ca873f (DEFAULT_TABLE table=test-table [id=5f89e720cf894d98ba9853cf4a14d151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:35.477201 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5bedbd8cba4140b866014052ca873f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:35.477811 2032 ts_tablet_manager.cc:1918] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:35.477909 2032 log.cc:1196] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/40f58e92bde0470f90e387c94d487076
I20260620 09:16:35.478188 2032 ts_tablet_manager.cc:1939] T 40f58e92bde0470f90e387c94d487076 P ce35eae8f9ae41bb98084d25ebfcd4f3: Deleting consensus metadata
I20260620 09:16:35.478587 2007 tablet_bootstrap.cc:492] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4: Bootstrap starting.
I20260620 09:16:35.478665 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5bedbd8cba4140b866014052ca873f (DEFAULT_TABLE table=test-table [id=5f89e720cf894d98ba9853cf4a14d151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:35.478796 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5bedbd8cba4140b866014052ca873f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:35.479071 2007 tablet_bootstrap.cc:654] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:35.480041 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5bedbd8cba4140b866014052ca873f (DEFAULT_TABLE table=test-table [id=5f89e720cf894d98ba9853cf4a14d151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:35.480146 2009 tablet_bootstrap.cc:492] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3: Bootstrap starting.
I20260620 09:16:35.480115 1490 catalog_manager.cc:4841] TS ce35eae8f9ae41bb98084d25ebfcd4f3 (127.0.98.2:43339): tablet 40f58e92bde0470f90e387c94d487076 (table test-table [id=934d096673cd41bdb7983249e27d2499]) successfully deleted
I20260620 09:16:35.480178 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5bedbd8cba4140b866014052ca873f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:35.480541 2007 tablet_bootstrap.cc:492] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4: No bootstrap required, opened a new log
I20260620 09:16:35.480540 2009 tablet_bootstrap.cc:654] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:35.480580 2007 ts_tablet_manager.cc:1397] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:16:35.480746 2007 raft_consensus.cc:348] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.480824 2007 raft_consensus.cc:374] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:35.480841 2007 raft_consensus.cc:729] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546536e7b2e24a46add30644ae2587b4, State: Initialized, Role: FOLLOWER
I20260620 09:16:35.480885 2007 consensus_queue.cc:260] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.480964 2007 ts_tablet_manager.cc:1428] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:35.481282 2009 tablet_bootstrap.cc:492] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3: No bootstrap required, opened a new log
I20260620 09:16:35.481320 2009 ts_tablet_manager.cc:1397] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:35.481444 2009 raft_consensus.cc:348] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.481493 2009 raft_consensus.cc:374] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:35.481504 2009 raft_consensus.cc:729] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce35eae8f9ae41bb98084d25ebfcd4f3, State: Initialized, Role: FOLLOWER
I20260620 09:16:35.481524 2008 tablet_bootstrap.cc:492] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b: Bootstrap starting.
I20260620 09:16:35.481537 2009 consensus_queue.cc:260] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.481622 2009 ts_tablet_manager.cc:1428] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:35.481951 2008 tablet_bootstrap.cc:654] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:35.482187 2013 raft_consensus.cc:482] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:35.482263 2013 raft_consensus.cc:504] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.482417 2030 raft_consensus.cc:2229] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:35.482549 2008 tablet_bootstrap.cc:492] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b: No bootstrap required, opened a new log
I20260620 09:16:35.482594 2008 ts_tablet_manager.cc:1397] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:35.482582 2013 leader_election.cc:288] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce35eae8f9ae41bb98084d25ebfcd4f3 (127.0.98.2:43339), 27bc63eea961489c8f121fc32fb3184b (127.0.98.3:43055)
I20260620 09:16:35.482697 2031 tablet_replica.cc:330] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4: stopping tablet replica
I20260620 09:16:35.482771 2030 raft_consensus.cc:2258] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:35.482976 2031 raft_consensus.cc:2229] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:35.483251 2031 raft_consensus.cc:2258] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:35.483754 2031 ts_tablet_manager.cc:1905] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:35.483752 2030 ts_tablet_manager.cc:1905] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:35.485142 2030 ts_tablet_manager.cc:1918] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:35.485231 2030 log.cc:1196] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/40f58e92bde0470f90e387c94d487076
I20260620 09:16:35.485397 2031 ts_tablet_manager.cc:1918] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:35.485472 2030 ts_tablet_manager.cc:1939] T 40f58e92bde0470f90e387c94d487076 P 27bc63eea961489c8f121fc32fb3184b: Deleting consensus metadata
I20260620 09:16:35.485497 2031 log.cc:1196] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/wal/wals/40f58e92bde0470f90e387c94d487076
I20260620 09:16:35.485812 2031 ts_tablet_manager.cc:1939] T 40f58e92bde0470f90e387c94d487076 P 546536e7b2e24a46add30644ae2587b4: Deleting consensus metadata
I20260620 09:16:35.482731 2008 raft_consensus.cc:348] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.485880 1491 catalog_manager.cc:4841] TS 27bc63eea961489c8f121fc32fb3184b (127.0.98.3:43055): tablet 40f58e92bde0470f90e387c94d487076 (table test-table [id=934d096673cd41bdb7983249e27d2499]) successfully deleted
I20260620 09:16:35.485909 2008 raft_consensus.cc:374] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:35.485942 2008 raft_consensus.cc:729] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27bc63eea961489c8f121fc32fb3184b, State: Initialized, Role: FOLLOWER
I20260620 09:16:35.485991 2008 consensus_queue.cc:260] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.486099 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.482709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43339 (local address 127.0.98.1:52863)
0620 09:16:35.482839 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.482852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.482908 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.483083 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.483084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.483086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:35.483148 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.483150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.484017 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.484018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.485151 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.485153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.485856 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.485857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.485935 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.485939 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.485988 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.486029 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 09:16:35.486102 2008 ts_tablet_manager.cc:1428] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:16:35.486328 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.482799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52863 (local address 127.0.98.2:43339)
0620 09:16:35.482881 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.482883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.482892 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:16:35.482986 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.482988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.482997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.483039 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:35.483219 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.483928 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.485247 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.485781 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.485857 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.485870 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.486235 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.486272 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 09:16:35.486531 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5bedbd8cba4140b866014052ca873f" candidate_uuid: "546536e7b2e24a46add30644ae2587b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" is_pre_election: true
I20260620 09:16:35.486651 1801 raft_consensus.cc:2454] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546536e7b2e24a46add30644ae2587b4 in term 0.
I20260620 09:16:35.486835 1594 leader_election.cc:302] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546536e7b2e24a46add30644ae2587b4, ce35eae8f9ae41bb98084d25ebfcd4f3; no voters:
I20260620 09:16:35.486945 2013 raft_consensus.cc:2781] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:35.486972 2013 raft_consensus.cc:482] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:35.486961 1490 catalog_manager.cc:4841] TS 546536e7b2e24a46add30644ae2587b4 (127.0.98.1:40489): tablet 40f58e92bde0470f90e387c94d487076 (table test-table [id=934d096673cd41bdb7983249e27d2499]) successfully deleted
I20260620 09:16:35.486991 2013 raft_consensus.cc:3037] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:35.487334 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.482946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57961 (local address 127.0.98.3:43055)
0620 09:16:35.482980 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.482982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.483112 (+ 130us) server_negotiation.cc:407] Connection header received
0620 09:16:35.483204 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.483205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.483213 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.483240 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:35.484074 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.484724 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.486486 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.487019 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.487088 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.487099 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.487245 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.487282 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:16:35.487376 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.482716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43055 (local address 127.0.98.1:57961)
0620 09:16:35.483037 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.483047 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.483138 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.483914 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.483915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.483917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:35.483991 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.483992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.484817 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.484818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.486394 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.486397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.487089 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.487089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.487164 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.487169 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.487230 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.487279 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":11,"thread_start_us":81,"threads_started":1}
I20260620 09:16:35.487643 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5bedbd8cba4140b866014052ca873f" candidate_uuid: "546536e7b2e24a46add30644ae2587b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bc63eea961489c8f121fc32fb3184b" is_pre_election: true
I20260620 09:16:35.487709 1941 raft_consensus.cc:2454] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546536e7b2e24a46add30644ae2587b4 in term 0.
I20260620 09:16:35.487691 2013 raft_consensus.cc:504] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.487820 2013 leader_election.cc:288] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [CANDIDATE]: Term 1 election: Requested vote from peers ce35eae8f9ae41bb98084d25ebfcd4f3 (127.0.98.2:43339), 27bc63eea961489c8f121fc32fb3184b (127.0.98.3:43055)
I20260620 09:16:35.488070 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5bedbd8cba4140b866014052ca873f" candidate_uuid: "546536e7b2e24a46add30644ae2587b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3"
I20260620 09:16:35.488133 1801 raft_consensus.cc:3037] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:35.488910 1801 raft_consensus.cc:2454] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 546536e7b2e24a46add30644ae2587b4 in term 1.
I20260620 09:16:35.489008 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5bedbd8cba4140b866014052ca873f" candidate_uuid: "546536e7b2e24a46add30644ae2587b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bc63eea961489c8f121fc32fb3184b"
I20260620 09:16:35.489063 1941 raft_consensus.cc:3037] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:35.489063 1594 leader_election.cc:302] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546536e7b2e24a46add30644ae2587b4, ce35eae8f9ae41bb98084d25ebfcd4f3; no voters:
W20260620 09:16:35.489188 1848 tablet.cc:2369] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:16:35.489322 2013 raft_consensus.cc:2781] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:35.489420 2013 raft_consensus.cc:686] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 1 LEADER]: Becoming Leader. State: Replica: 546536e7b2e24a46add30644ae2587b4, State: Running, Role: LEADER
I20260620 09:16:35.489471 2013 consensus_queue.cc:237] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.490001 1504 catalog_manager.cc:5495] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 546536e7b2e24a46add30644ae2587b4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "546536e7b2e24a46add30644ae2587b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } health_report { overall_health: UNKNOWN } } }
I20260620 09:16:35.490191 1941 raft_consensus.cc:2454] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 546536e7b2e24a46add30644ae2587b4 in term 1.
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:35.515455 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.508448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50380 (local address 127.0.98.62:44121)
0620 09:16:35.508518 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.508521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.508616 (+ 95us) server_negotiation.cc:407] Connection header received
0620 09:16:35.508688 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.508689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.508699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.508756 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:35.509334 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.509831 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.510854 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.511026 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.511070 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.513274 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:35.513276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:35.513276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:35.513283 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:35.514464 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.514785 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.514786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.514787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.514832 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.514981 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.514982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.514982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.515082 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:35.515138 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.515273 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.515350 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:16:35.515525 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.508374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44121 (local address 127.0.0.1:50380)
0620 09:16:35.508536 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.508548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.508627 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.508773 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.508773 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.509172 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:35.509230 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.509243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.509901 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.509902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.510784 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.510787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.511083 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.511084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.511139 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.511711 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:35.511718 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:35.513197 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:35.514535 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.514542 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.514545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.514725 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.514881 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:35.514882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:35.514882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:35.514936 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:35.515136 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:35.515143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:35.515190 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.515248 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.515415 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":69,"thread_start_us":103,"threads_started":1}
I20260620 09:16:35.517576 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50380:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 09:16:35.517765 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:35.519042 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 5cff1864dabd4a23aad0f1ddf5005637 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=30266e6802ee4f97ad73e53f85d372a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:16:35.519052 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 5cff1864dabd4a23aad0f1ddf5005637 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=30266e6802ee4f97ad73e53f85d372a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:16:35.519061 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 5cff1864dabd4a23aad0f1ddf5005637 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=30266e6802ee4f97ad73e53f85d372a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:16:35.519192 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cff1864dabd4a23aad0f1ddf5005637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:35.519198 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cff1864dabd4a23aad0f1ddf5005637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:35.519198 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cff1864dabd4a23aad0f1ddf5005637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:35.520635 2009 tablet_bootstrap.cc:492] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3: Bootstrap starting.
I20260620 09:16:35.520638 2007 tablet_bootstrap.cc:492] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4: Bootstrap starting.
I20260620 09:16:35.520689 2008 tablet_bootstrap.cc:492] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: Bootstrap starting.
I20260620 09:16:35.521008 2007 tablet_bootstrap.cc:654] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:35.521008 2009 tablet_bootstrap.cc:654] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:35.521091 2008 tablet_bootstrap.cc:654] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:35.521906 2007 tablet_bootstrap.cc:492] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4: No bootstrap required, opened a new log
I20260620 09:16:35.521955 2007 ts_tablet_manager.cc:1397] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:35.522068 2009 tablet_bootstrap.cc:492] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3: No bootstrap required, opened a new log
I20260620 09:16:35.522109 2009 ts_tablet_manager.cc:1397] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:35.522218 2008 tablet_bootstrap.cc:492] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: No bootstrap required, opened a new log
I20260620 09:16:35.522210 2007 raft_consensus.cc:348] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.522254 2008 ts_tablet_manager.cc:1397] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:35.522226 2009 raft_consensus.cc:348] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.522277 2007 raft_consensus.cc:374] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:35.522298 2007 raft_consensus.cc:729] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546536e7b2e24a46add30644ae2587b4, State: Initialized, Role: FOLLOWER
I20260620 09:16:35.522298 2009 raft_consensus.cc:374] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:35.522318 2009 raft_consensus.cc:729] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce35eae8f9ae41bb98084d25ebfcd4f3, State: Initialized, Role: FOLLOWER
I20260620 09:16:35.522341 2007 consensus_queue.cc:260] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.522359 2009 consensus_queue.cc:260] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.522385 2008 raft_consensus.cc:348] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.522418 2007 ts_tablet_manager.cc:1428] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:35.522440 2008 raft_consensus.cc:374] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:35.522446 2009 ts_tablet_manager.cc:1428] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:35.522459 2008 raft_consensus.cc:729] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27bc63eea961489c8f121fc32fb3184b, State: Initialized, Role: FOLLOWER
I20260620 09:16:35.522496 2008 consensus_queue.cc:260] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.522521 2013 tablet_replica.cc:439] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } } }
I20260620 09:16:35.522565 2008 ts_tablet_manager.cc:1428] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:35.522575 2013 tablet_replica.cc:442] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:35.522710 2014 tablet_replica.cc:439] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } } }
I20260620 09:16:35.522691 2025 tablet_replica.cc:439] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } } }
I20260620 09:16:35.522776 2014 tablet_replica.cc:442] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:35.522776 2025 tablet_replica.cc:442] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 09:16:35.588240 1708 tablet.cc:2369] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:16:35.609120 2014 raft_consensus.cc:482] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:35.609223 2014 raft_consensus.cc:504] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.609645 2014 leader_election.cc:288] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 546536e7b2e24a46add30644ae2587b4 (127.0.98.1:40489), ce35eae8f9ae41bb98084d25ebfcd4f3 (127.0.98.2:43339)
I20260620 09:16:35.612733 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.609908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41031 (local address 127.0.98.1:40489)
0620 09:16:35.609959 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.609961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.609993 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:16:35.610023 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.610023 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.610037 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.610088 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:35.610273 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.610781 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.611940 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.612418 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.612467 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.612477 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.612655 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.612681 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:16:35.612774 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.609820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40489 (local address 127.0.98.3:41031)
0620 09:16:35.609872 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.609883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.609950 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.610124 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.610125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.610126 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:35.610218 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.610220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.610846 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.610847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.611863 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.611866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.612495 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.612496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.612575 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.612579 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.612653 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.612691 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 09:16:35.613018 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cff1864dabd4a23aad0f1ddf5005637" candidate_uuid: "27bc63eea961489c8f121fc32fb3184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546536e7b2e24a46add30644ae2587b4" is_pre_election: true
I20260620 09:16:35.613121 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.609909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43339 (local address 127.0.98.3:38553)
0620 09:16:35.610024 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.610033 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.610109 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.610252 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.610253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.610254 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:35.610338 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.610339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.610954 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.610955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.612232 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.612234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.612890 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.612891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.612964 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.612968 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.613021 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.613054 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260620 09:16:35.613127 1661 raft_consensus.cc:2454] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27bc63eea961489c8f121fc32fb3184b in term 0.
I20260620 09:16:35.613131 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.609987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38553 (local address 127.0.98.2:43339)
0620 09:16:35.610035 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.610037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.610096 (+ 59us) server_negotiation.cc:407] Connection header received
0620 09:16:35.610170 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.610171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.610183 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.610236 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:35.610400 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.610867 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.612332 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.612808 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.612887 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.612897 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.613035 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.613072 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:16:35.613319 1874 leader_election.cc:302] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27bc63eea961489c8f121fc32fb3184b, 546536e7b2e24a46add30644ae2587b4; no voters:
I20260620 09:16:35.613363 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cff1864dabd4a23aad0f1ddf5005637" candidate_uuid: "27bc63eea961489c8f121fc32fb3184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" is_pre_election: true
I20260620 09:16:35.613442 2014 raft_consensus.cc:2781] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:35.613444 1801 raft_consensus.cc:2454] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27bc63eea961489c8f121fc32fb3184b in term 0.
I20260620 09:16:35.613477 2014 raft_consensus.cc:482] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:35.613495 2014 raft_consensus.cc:3037] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:35.614156 2014 raft_consensus.cc:504] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.614269 2014 leader_election.cc:288] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [CANDIDATE]: Term 1 election: Requested vote from peers 546536e7b2e24a46add30644ae2587b4 (127.0.98.1:40489), ce35eae8f9ae41bb98084d25ebfcd4f3 (127.0.98.2:43339)
I20260620 09:16:35.614416 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cff1864dabd4a23aad0f1ddf5005637" candidate_uuid: "27bc63eea961489c8f121fc32fb3184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3"
I20260620 09:16:35.614413 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cff1864dabd4a23aad0f1ddf5005637" candidate_uuid: "27bc63eea961489c8f121fc32fb3184b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546536e7b2e24a46add30644ae2587b4"
I20260620 09:16:35.614481 1661 raft_consensus.cc:3037] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:35.614483 1801 raft_consensus.cc:3037] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:35.615015 1801 raft_consensus.cc:2454] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27bc63eea961489c8f121fc32fb3184b in term 1.
I20260620 09:16:35.615164 1874 leader_election.cc:302] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27bc63eea961489c8f121fc32fb3184b, ce35eae8f9ae41bb98084d25ebfcd4f3; no voters:
I20260620 09:16:35.615242 2014 raft_consensus.cc:2781] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:35.615306 1661 raft_consensus.cc:2454] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27bc63eea961489c8f121fc32fb3184b in term 1.
I20260620 09:16:35.615389 2014 raft_consensus.cc:686] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [term 1 LEADER]: Becoming Leader. State: Replica: 27bc63eea961489c8f121fc32fb3184b, State: Running, Role: LEADER
I20260620 09:16:35.615470 2014 consensus_queue.cc:237] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } }
I20260620 09:16:35.615885 2049 tablet_replica.cc:439] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: TxnStatusTablet state changed. Reason: New leader 27bc63eea961489c8f121fc32fb3184b. Latest consensus state: current_term: 1 leader_uuid: "27bc63eea961489c8f121fc32fb3184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } } }
I20260620 09:16:35.615968 2049 tablet_replica.cc:442] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:16:35.616129 2051 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 09:16:35.616233 2051 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 09:16:35.616201 1504 catalog_manager.cc:5495] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b reported cstate change: term changed from 0 to 1, leader changed from <none> to 27bc63eea961489c8f121fc32fb3184b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "27bc63eea961489c8f121fc32fb3184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } health_report { overall_health: HEALTHY } } }
W20260620 09:16:35.629145 1988 tablet.cc:2369] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:16:35.639370 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.636566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43055 (local address 127.0.0.1:38514)
0620 09:16:35.636652 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.636664 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.636721 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.636893 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.636894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.636895 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:16:35.636942 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.636945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.637825 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.637826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.638398 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.638401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.638850 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.638852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.638906 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.638922 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:16:35.639209 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:16:35.639220 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.639265 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.639312 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 09:16:35.639374 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.636621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.0.98.3:43055)
0620 09:16:35.636673 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.636675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.636703 (+ 28us) server_negotiation.cc:407] Connection header received
0620 09:16:35.636791 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.636792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.636822 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.636893 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:35.636989 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.637735 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.638471 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.638776 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.638837 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.638999 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:35.639128 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:35.639195 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.639278 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.639295 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:16:35.640496 1661 raft_consensus.cc:1261] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4 [term 1 FOLLOWER]: Refusing update from remote peer 27bc63eea961489c8f121fc32fb3184b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:35.640563 1801 raft_consensus.cc:1261] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 1 FOLLOWER]: Refusing update from remote peer 27bc63eea961489c8f121fc32fb3184b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:35.640810 2049 consensus_queue.cc:1035] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [LEADER]: Connected to new peer: Peer: permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:35.640938 2014 consensus_queue.cc:1035] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:35.641496 2013 tablet_replica.cc:439] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4: TxnStatusTablet state changed. Reason: New leader 27bc63eea961489c8f121fc32fb3184b. Latest consensus state: current_term: 1 leader_uuid: "27bc63eea961489c8f121fc32fb3184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } } }
I20260620 09:16:35.641584 2013 tablet_replica.cc:442] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:35.641566 2025 tablet_replica.cc:439] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3: TxnStatusTablet state changed. Reason: New leader 27bc63eea961489c8f121fc32fb3184b. Latest consensus state: current_term: 1 leader_uuid: "27bc63eea961489c8f121fc32fb3184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } } }
I20260620 09:16:35.641646 2025 tablet_replica.cc:442] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:35.642387 2014 tablet_replica.cc:439] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27bc63eea961489c8f121fc32fb3184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } } }
I20260620 09:16:35.642478 2014 tablet_replica.cc:442] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:16:35.642400 2056 tablet_replica.cc:439] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27bc63eea961489c8f121fc32fb3184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } } }
I20260620 09:16:35.642549 2056 tablet_replica.cc:442] T 5cff1864dabd4a23aad0f1ddf5005637 P 27bc63eea961489c8f121fc32fb3184b: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:16:35.642869 2013 tablet_replica.cc:439] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27bc63eea961489c8f121fc32fb3184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } } }
I20260620 09:16:35.642946 2013 tablet_replica.cc:442] T 5cff1864dabd4a23aad0f1ddf5005637 P 546536e7b2e24a46add30644ae2587b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:35.643476 2025 tablet_replica.cc:439] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27bc63eea961489c8f121fc32fb3184b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546536e7b2e24a46add30644ae2587b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40489 } } peers { permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 } } peers { permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 } } }
I20260620 09:16:35.643559 2025 tablet_replica.cc:442] T 5cff1864dabd4a23aad0f1ddf5005637 P ce35eae8f9ae41bb98084d25ebfcd4f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:35.646678 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.644350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.0.98.1:40489)
0620 09:16:35.644389 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.644391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.644398 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:35.644458 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.644458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.644473 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.644532 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:35.644650 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.645081 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.646083 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.646252 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.646291 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.646427 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:35.646512 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:35.646555 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.646607 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.646620 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:16:35.651180 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.648722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43697 (local address 127.0.98.3:43055)
0620 09:16:35.648759 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.648761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.648891 (+ 130us) server_negotiation.cc:407] Connection header received
0620 09:16:35.648915 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.648916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.648924 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.648986 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:35.649071 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.649703 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.650399 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.650693 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.650755 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.650952 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:35.650982 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:35.651032 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.651101 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.651116 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:16:35.651187 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.648661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43055 (local address 127.0.98.1:43697)
0620 09:16:35.648826 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.648834 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.648885 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.648988 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.648988 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.648989 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:16:35.649027 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.649028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.649796 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.649797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.650332 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.650333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.650790 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.650791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.650841 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.650853 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:16:35.651052 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:16:35.651054 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.651089 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.651128 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260620 09:16:35.653049 1941 raft_consensus.cc:1261] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b [term 1 FOLLOWER]: Refusing update from remote peer 546536e7b2e24a46add30644ae2587b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:35.653062 1801 raft_consensus.cc:1261] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 1 FOLLOWER]: Refusing update from remote peer 546536e7b2e24a46add30644ae2587b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:35.653272 2013 consensus_queue.cc:1035] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce35eae8f9ae41bb98084d25ebfcd4f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:35.653414 2037 consensus_queue.cc:1035] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bc63eea961489c8f121fc32fb3184b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:35.660967 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.658641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42431 (local address 127.0.98.1:40489)
0620 09:16:35.658685 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.658687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.658695 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:35.658713 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.658714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.658723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.658777 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:35.658879 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.659362 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.660409 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.660591 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.660638 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.660783 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:35.660806 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:35.660836 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.660889 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.660901 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:16:35.661010 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.658556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40489 (local address 127.0.98.3:42431)
0620 09:16:35.658618 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:35.658629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:35.658665 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:35.658780 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:35.658781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:35.658782 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:16:35.658830 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.658831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.659425 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.659426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.660338 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:35.660341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.660641 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.660642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:35.660703 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.660717 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:16:35.660840 (+ 123us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:16:35.660841 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 09:16:35.660886 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:16:35.660942 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 09:16:35.668733 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:35.668828 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:35.669386 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 445fbe4e95394a97b5c192d5a891d97f: Sending DeleteTablet for 3 replicas of tablet 2e5bedbd8cba4140b866014052ca873f
I20260620 09:16:35.669625 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e5bedbd8cba4140b866014052ca873f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38500
I20260620 09:16:35.669720 2030 tablet_replica.cc:330] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b: stopping tablet replica
I20260620 09:16:35.669804 2030 raft_consensus.cc:2229] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:35.669842 2030 raft_consensus.cc:2258] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:35.670143 2030 ts_tablet_manager.cc:1905] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:35.670270 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e5bedbd8cba4140b866014052ca873f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36516
I20260620 09:16:35.670397 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e5bedbd8cba4140b866014052ca873f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57798
I20260620 09:16:35.670480 2031 tablet_replica.cc:330] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4: stopping tablet replica
I20260620 09:16:35.670537 2031 raft_consensus.cc:2229] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:35.670617 2031 raft_consensus.cc:2258] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:35.670886 2031 ts_tablet_manager.cc:1905] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:35.671607 2032 tablet_replica.cc:330] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3: stopping tablet replica
I20260620 09:16:35.671685 2032 raft_consensus.cc:2229] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:35.671721 2032 raft_consensus.cc:2258] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:35.671720 2030 ts_tablet_manager.cc:1918] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:16:35.671768 2030 log.cc:1196] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/2e5bedbd8cba4140b866014052ca873f
I20260620 09:16:35.671967 2030 ts_tablet_manager.cc:1939] T 2e5bedbd8cba4140b866014052ca873f P 27bc63eea961489c8f121fc32fb3184b: Deleting consensus metadata
I20260620 09:16:35.672240 1491 catalog_manager.cc:4841] TS 27bc63eea961489c8f121fc32fb3184b (127.0.98.3:43055): tablet 2e5bedbd8cba4140b866014052ca873f (table test-table [id=5f89e720cf894d98ba9853cf4a14d151]) successfully deleted
I20260620 09:16:35.672329 2031 ts_tablet_manager.cc:1918] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:16:35.672382 2031 log.cc:1196] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-0/wal/wals/2e5bedbd8cba4140b866014052ca873f
I20260620 09:16:35.672564 2031 ts_tablet_manager.cc:1939] T 2e5bedbd8cba4140b866014052ca873f P 546536e7b2e24a46add30644ae2587b4: Deleting consensus metadata
I20260620 09:16:35.672847 2032 ts_tablet_manager.cc:1905] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:35.672927 1490 catalog_manager.cc:4841] TS 546536e7b2e24a46add30644ae2587b4 (127.0.98.1:40489): tablet 2e5bedbd8cba4140b866014052ca873f (table test-table [id=5f89e720cf894d98ba9853cf4a14d151]) successfully deleted
I20260620 09:16:35.674343 2032 ts_tablet_manager.cc:1918] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:16:35.674389 2032 log.cc:1196] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/2e5bedbd8cba4140b866014052ca873f
I20260620 09:16:35.674571 2032 ts_tablet_manager.cc:1939] T 2e5bedbd8cba4140b866014052ca873f P ce35eae8f9ae41bb98084d25ebfcd4f3: Deleting consensus metadata
I20260620 09:16:35.674867 1490 catalog_manager.cc:4841] TS ce35eae8f9ae41bb98084d25ebfcd4f3 (127.0.98.2:43339): tablet 2e5bedbd8cba4140b866014052ca873f (table test-table [id=5f89e720cf894d98ba9853cf4a14d151]) successfully deleted
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:16:35.689611 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.675889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.0.98.1:40489)
0620 09:16:35.675927 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.675928 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.679826 (+ 3898us) server_negotiation.cc:407] Connection header received
0620 09:16:35.679882 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.679883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.679895 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.679963 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:35.680653 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.681426 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.682468 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.682815 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.682880 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.686311 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:35.686320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:35.686321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:35.686329 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:35.688355 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.688769 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.688772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.688772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.688836 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.689015 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.689015 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.689016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.689186 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:35.689320 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.689406 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.689490 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 09:16:35.690171 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57816
I20260620 09:16:35.699162 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:35.691172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.0.98.62:44121)
0620 09:16:35.691233 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:35.691235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:35.691494 (+ 259us) server_negotiation.cc:407] Connection header received
0620 09:16:35.691545 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:35.691546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:35.691558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:35.691612 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:35.692264 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.693073 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.694101 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:35.694377 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:35.694423 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:35.696050 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:35.696051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:35.696052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:35.696060 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:35.697867 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.698272 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.698273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.698274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.698349 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:35.698557 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:35.698559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:35.698559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:35.698754 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:35.698864 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:35.698986 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 09:16:35.699072 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 09:16:35.699378 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50386
I20260620 09:16:35.700285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 1576
I20260620 09:16:35.707159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 1716
I20260620 09:16:35.712716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 1856
I20260620 09:16:35.716985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 1472
2026-06-20T09:16:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (1975 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[2072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[2072](info): set up 2 sockets
Jun 20 09:16:35 dist-test-slave-rq8c krb5kdc[2072](info): commencing operation
krb5kdc: starting...
W20260620 09:16:36.667173 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:16:36 dist-test-slave-rq8c krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:16:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:16:36.775727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33227
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:36.853515 2085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:36.853706 2085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:36.853726 2085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:36.854791 2085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:36.854825 2085 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 09:16:36.854835 2085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:36.854847 2085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:36.854857 2085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:36.854874 2085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:36.856266 2085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33227
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:36.856447 2085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:36.856617 2085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:36.861057 2091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:36.861063 2093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:36.861244 2085 server_base.cc:1034] running on GCE node
W20260620 09:16:36.861040 2090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:36.861575 2085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:36.861871 2085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:36.863008 2085 hybrid_clock.cc:648] HybridClock initialized: now 1781946996862993 us; error 25 us; skew 500 ppm
Jun 20 09:16:36 dist-test-slave-rq8c krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:36.865309 2085 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:16:36.865708 2085 webserver.cc:460] Webserver started at http://127.0.98.62:46077/ using document root <none> and password file <none>
I20260620 09:16:36.865909 2085 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:36.865952 2085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:36.866070 2085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:36.867069 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "04caaa4ce9d7457c887cb18ba3ac60d4"
format_stamp: "Formatted at 2026-06-20 09:16:36 on dist-test-slave-rq8c"
server_key: "ed271137c3ff70d4554a39776729dde3"
server_key_iv: "e72996e4d78d0c27f3ae8da38cd16497"
server_key_version: "encryptionkey@0"
I20260620 09:16:36.867384 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "04caaa4ce9d7457c887cb18ba3ac60d4"
format_stamp: "Formatted at 2026-06-20 09:16:36 on dist-test-slave-rq8c"
server_key: "ed271137c3ff70d4554a39776729dde3"
server_key_iv: "e72996e4d78d0c27f3ae8da38cd16497"
server_key_version: "encryptionkey@0"
I20260620 09:16:36.868659 2085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:36.869400 2100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:36.869554 2085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:36.869629 2085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "04caaa4ce9d7457c887cb18ba3ac60d4"
format_stamp: "Formatted at 2026-06-20 09:16:36 on dist-test-slave-rq8c"
server_key: "ed271137c3ff70d4554a39776729dde3"
server_key_iv: "e72996e4d78d0c27f3ae8da38cd16497"
server_key_version: "encryptionkey@0"
I20260620 09:16:36.869705 2085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:36.899698 2085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:36.901510 2085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:36.901618 2085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:36.905997 2085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33227
I20260620 09:16:36.906033 2162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33227 every 8 connection(s)
I20260620 09:16:36.906373 2085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:36.906988 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:36.909220 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4: Bootstrap starting.
I20260620 09:16:36.909751 2163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:36.909996 2163 log.cc:824] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:36.910264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 2085
I20260620 09:16:36.910356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:36.910518 392 external_mini_cluster.cc:1428] Setting key c70d3b1de9d55afe7f60135d4d03f7c9
I20260620 09:16:36.910732 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4: No bootstrap required, opened a new log
I20260620 09:16:36.913038 2163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04caaa4ce9d7457c887cb18ba3ac60d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33227 } }
I20260620 09:16:36.913172 2163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:36.913204 2163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04caaa4ce9d7457c887cb18ba3ac60d4, State: Initialized, Role: FOLLOWER
I20260620 09:16:36.913327 2163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04caaa4ce9d7457c887cb18ba3ac60d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33227 } }
I20260620 09:16:36.913399 2163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:36.913437 2163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:36.913482 2163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:36.913978 2163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04caaa4ce9d7457c887cb18ba3ac60d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33227 } }
I20260620 09:16:36.914069 2163 leader_election.cc:302] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04caaa4ce9d7457c887cb18ba3ac60d4; no voters:
I20260620 09:16:36.914212 2163 leader_election.cc:288] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:36.914250 2168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:36.914358 2168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [term 1 LEADER]: Becoming Leader. State: Replica: 04caaa4ce9d7457c887cb18ba3ac60d4, State: Running, Role: LEADER
I20260620 09:16:36.914474 2163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:36.914577 2168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04caaa4ce9d7457c887cb18ba3ac60d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33227 } }
I20260620 09:16:36.914875 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04caaa4ce9d7457c887cb18ba3ac60d4. Latest consensus state: current_term: 1 leader_uuid: "04caaa4ce9d7457c887cb18ba3ac60d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04caaa4ce9d7457c887cb18ba3ac60d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33227 } } }
I20260620 09:16:36.915030 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:36.915477 2177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:36.915833 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04caaa4ce9d7457c887cb18ba3ac60d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04caaa4ce9d7457c887cb18ba3ac60d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33227 } } }
I20260620 09:16:36.915911 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:36.916002 2177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 09:16:36 dist-test-slave-rq8c krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:36.917634 2177 catalog_manager.cc:1347] Generated new cluster ID: 4e62294515b4430e926275fb36417044
I20260620 09:16:36.917685 2177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:16:36.920902 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:36.910970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.0.98.62:33227)
0620 09:16:36.911142 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:36.911150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:36.911173 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:16:36.911923 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:36.911930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:36.911982 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:36.912105 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:36.912666 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:36.913648 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:36.914532 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:36.915095 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:36.915195 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:36.917952 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:36.917956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:36.917956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:36.917967 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:36.919503 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:36.919869 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:36.919872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:36.919873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:36.919954 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:36.920118 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:36.920119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:36.920119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:36.920341 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:36.920498 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:36.920563 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 09:16:36.920691 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260620 09:16:36.934314 2177 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:36.934823 2177 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:36.938185 2177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4: Generated new TSK 0
I20260620 09:16:36.938354 2177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:16:36.964525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33227
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:37.044016 2189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:37.044245 2189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:37.044274 2189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:37.045543 2189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:37.045607 2189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:37.045629 2189 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 09:16:37.045639 2189 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:16:37.045656 2189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:37.047216 2189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:37.047482 2189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:37.047760 2189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:37.052827 2197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:37.052842 2194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:37.052944 2189 server_base.cc:1034] running on GCE node
W20260620 09:16:37.052842 2195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:37.053226 2189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:37.053468 2189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:37.054670 2189 hybrid_clock.cc:648] HybridClock initialized: now 1781946997054643 us; error 42 us; skew 500 ppm
Jun 20 09:16:37 dist-test-slave-rq8c krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:37.057078 2189 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:16:37.057523 2189 webserver.cc:460] Webserver started at http://127.0.98.1:38035/ using document root <none> and password file <none>
I20260620 09:16:37.057741 2189 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:37.057802 2189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:37.057894 2189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:37.058909 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "f8c5b69d877541229586cb7f981a6d1e"
format_stamp: "Formatted at 2026-06-20 09:16:37 on dist-test-slave-rq8c"
server_key: "0ea38ea734d98a3816aba0b4c5d7d0a5"
server_key_iv: "85d5d5da11aec9c144bb9aad5ba3dca3"
server_key_version: "encryptionkey@0"
I20260620 09:16:37.059211 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f8c5b69d877541229586cb7f981a6d1e"
format_stamp: "Formatted at 2026-06-20 09:16:37 on dist-test-slave-rq8c"
server_key: "0ea38ea734d98a3816aba0b4c5d7d0a5"
server_key_iv: "85d5d5da11aec9c144bb9aad5ba3dca3"
server_key_version: "encryptionkey@0"
I20260620 09:16:37.060482 2189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:37.061180 2204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:37.061414 2189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:37.061515 2189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "f8c5b69d877541229586cb7f981a6d1e"
format_stamp: "Formatted at 2026-06-20 09:16:37 on dist-test-slave-rq8c"
server_key: "0ea38ea734d98a3816aba0b4c5d7d0a5"
server_key_iv: "85d5d5da11aec9c144bb9aad5ba3dca3"
server_key_version: "encryptionkey@0"
I20260620 09:16:37.061575 2189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:37.074595 2189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:37.076577 2189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:37.076685 2189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:37.077117 2189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:37.077155 2189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:37.077178 2189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:37.077189 2189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:16:37 dist-test-slave-rq8c krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:37.083602 2189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40901
I20260620 09:16:37.083977 2319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40901 every 8 connection(s)
I20260620 09:16:37.084012 2189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:16:37.086001 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.077603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45039 (local address 127.0.98.62:33227)
0620 09:16:37.077643 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.077645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.077872 (+ 227us) server_negotiation.cc:407] Connection header received
0620 09:16:37.078408 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.078409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.078423 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.078503 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:37.079293 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.080090 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.081100 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.081306 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.081363 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.083352 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:37.083354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:37.083354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:37.083361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:37.084832 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.085226 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.085227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.085227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.085294 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.085486 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.085487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.085487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.085589 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:37.085641 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.085835 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.085933 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:16:37.086095 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.077603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33227 (local address 127.0.98.1:45039)
0620 09:16:37.077764 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.077797 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.078289 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.078566 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.078570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.078895 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:37.079191 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.079197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.080187 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.080188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.081017 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.081019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.081391 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.081392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.081501 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.081959 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:37.081973 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:37.083267 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:37.084905 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.084910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.084912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.085130 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.085358 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.085358 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.085358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.085447 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.085630 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:37.085632 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:37.085742 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.085813 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.085925 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260620 09:16:37.086548 2320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33227
I20260620 09:16:37.086621 2320 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:37.086772 2320 heartbeater.cc:511] Master 127.0.98.62:33227 requested a full tablet report, sending...
I20260620 09:16:37.087292 2116 ts_manager.cc:194] Registered new tserver with Master: f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901)
I20260620 09:16:37.087996 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45039
I20260620 09:16:37.089246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 2189
I20260620 09:16:37.089341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:16:37.089542 392 external_mini_cluster.cc:1428] Setting key 2489a48d1ef3a0123c818a9eeffdfa8f
I20260620 09:16:37.093075 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.087549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53889 (local address 127.0.98.62:33227)
0620 09:16:37.087585 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.087587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.087739 (+ 152us) server_negotiation.cc:407] Connection header received
0620 09:16:37.087815 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.087816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.087827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.087881 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:37.088207 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.088649 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.089484 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.089701 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.089761 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.090650 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:37.090651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:37.090651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:37.090657 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:37.091961 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.092289 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.092290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.092291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.092360 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.092570 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.092571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.092571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.092686 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:37.092741 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.092920 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.092990 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:16:37.093082 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.087480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33227 (local address 127.0.98.1:53889)
0620 09:16:37.087673 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.087684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.087755 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.087892 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.087893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.088068 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:37.088141 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.088143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.088730 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.088731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.089397 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.089400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.089778 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.089779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.089846 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.090294 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:37.090298 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:37.090571 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:37.092031 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.092031 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.092032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.092233 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.092429 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.092430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.092430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.092490 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.092773 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:37.092774 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:37.092818 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.092881 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.092978 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:16:37.115593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33227
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:37.195030 2329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:37.195232 2329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:37.195255 2329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:37.196404 2329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:37.196444 2329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:37.196463 2329 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 09:16:37.196476 2329 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:16:37.196493 2329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:37.197988 2329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:37.198192 2329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:37.198395 2329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:37.202992 2335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:37.203125 2329 server_base.cc:1034] running on GCE node
W20260620 09:16:37.203022 2334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:37.203013 2337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:37.203436 2329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:37.203680 2329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:37.204818 2329 hybrid_clock.cc:648] HybridClock initialized: now 1781946997204804 us; error 31 us; skew 500 ppm
Jun 20 09:16:37 dist-test-slave-rq8c krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:37.207203 2329 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:16:37.207670 2329 webserver.cc:460] Webserver started at http://127.0.98.2:37397/ using document root <none> and password file <none>
I20260620 09:16:37.207885 2329 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:37.207931 2329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:37.208058 2329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:37.209095 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "30cf23263eda4efaaac4be509e733248"
format_stamp: "Formatted at 2026-06-20 09:16:37 on dist-test-slave-rq8c"
server_key: "9fe8427f0d684640cdac0be75395af87"
server_key_iv: "83719275d7a32c7736d69e98255cddba"
server_key_version: "encryptionkey@0"
I20260620 09:16:37.209446 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "30cf23263eda4efaaac4be509e733248"
format_stamp: "Formatted at 2026-06-20 09:16:37 on dist-test-slave-rq8c"
server_key: "9fe8427f0d684640cdac0be75395af87"
server_key_iv: "83719275d7a32c7736d69e98255cddba"
server_key_version: "encryptionkey@0"
I20260620 09:16:37.210713 2329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:16:37.211452 2344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:37.211652 2329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:37.211733 2329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "30cf23263eda4efaaac4be509e733248"
format_stamp: "Formatted at 2026-06-20 09:16:37 on dist-test-slave-rq8c"
server_key: "9fe8427f0d684640cdac0be75395af87"
server_key_iv: "83719275d7a32c7736d69e98255cddba"
server_key_version: "encryptionkey@0"
I20260620 09:16:37.211800 2329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:37.224057 2329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:37.226011 2329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:37.226159 2329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:37.226720 2329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:37.226769 2329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:37.226799 2329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:37.226828 2329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:16:37 dist-test-slave-rq8c krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:37.233148 2329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45001
I20260620 09:16:37.233219 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45001 every 8 connection(s)
I20260620 09:16:37.233489 2329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:16:37.235733 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.227101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55067 (local address 127.0.98.62:33227)
0620 09:16:37.227160 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.227162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.227366 (+ 204us) server_negotiation.cc:407] Connection header received
0620 09:16:37.227906 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.227907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.227919 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.227964 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:37.228568 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.229110 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.230126 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.230413 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.230487 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.233126 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:37.233127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:37.233128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:37.233136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:37.234580 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.234996 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.234996 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.234997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.235042 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.235212 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.235213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.235213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.235302 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:37.235346 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.235558 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.235646 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:16:37.235941 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.227089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33227 (local address 127.0.98.2:55067)
0620 09:16:37.227244 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.227284 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.227814 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.228033 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.228037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.228297 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:37.228485 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.228490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.229193 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.229194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.230059 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.230062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.230541 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.230542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.230718 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.231253 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:37.231262 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:37.233033 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:37.234669 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.234675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.234676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.234932 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.235081 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.235081 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.235082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.235170 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.235348 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:37.235350 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:37.235463 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.235544 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.235724 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260620 09:16:37.236378 2460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33227
I20260620 09:16:37.236454 2460 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:37.236624 2460 heartbeater.cc:511] Master 127.0.98.62:33227 requested a full tablet report, sending...
I20260620 09:16:37.236960 2117 ts_manager.cc:194] Registered new tserver with Master: 30cf23263eda4efaaac4be509e733248 (127.0.98.2:45001)
I20260620 09:16:37.237414 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55067
I20260620 09:16:37.240132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 2329
I20260620 09:16:37.240211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:16:37.240371 392 external_mini_cluster.cc:1428] Setting key b5c2685527426c6ae78621cd79bf85ad
I20260620 09:16:37.242800 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.237320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60905 (local address 127.0.98.62:33227)
0620 09:16:37.237356 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.237358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.237461 (+ 103us) server_negotiation.cc:407] Connection header received
0620 09:16:37.237522 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.237522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.237531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.237584 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:37.237980 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.238454 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.239147 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.239343 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.239402 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.240298 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:37.240300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:37.240300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:37.240307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:37.241691 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.242010 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.242011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.242011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.242069 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.242280 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.242281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.242281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.242428 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:37.242492 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.242634 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.242711 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:16:37.242815 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.237257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33227 (local address 127.0.98.2:60905)
0620 09:16:37.237394 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.237404 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.237467 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.237695 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.237696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.237842 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:37.237902 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.237903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.238547 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.238548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.239073 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.239075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.239402 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.239403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.239477 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.239890 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:37.239895 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:37.240187 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:37.241781 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.241782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.241783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.241947 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.242132 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.242133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.242133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.242215 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.242491 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:37.242492 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:37.242536 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.242605 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.242706 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:16:37.261801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33227
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:37.339668 2469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:37.339874 2469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:37.339902 2469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:37.341035 2469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:37.341075 2469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:37.341094 2469 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 09:16:37.341105 2469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:16:37.341120 2469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:37.342489 2469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:37.342664 2469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:37.342846 2469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:37.347458 2474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:37.347499 2469 server_base.cc:1034] running on GCE node
W20260620 09:16:37.347468 2477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:37.347556 2475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:37.347862 2469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:37.348068 2469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:37.349210 2469 hybrid_clock.cc:648] HybridClock initialized: now 1781946997349201 us; error 35 us; skew 500 ppm
Jun 20 09:16:37 dist-test-slave-rq8c krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:37.351471 2469 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:16:37.351913 2469 webserver.cc:460] Webserver started at http://127.0.98.3:36453/ using document root <none> and password file <none>
I20260620 09:16:37.352118 2469 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:37.352161 2469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:37.352278 2469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:37.353279 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "56ff42249922481ea6ced5ce70352b86"
format_stamp: "Formatted at 2026-06-20 09:16:37 on dist-test-slave-rq8c"
server_key: "76af7ade1bd12bde068723cea5e76ceb"
server_key_iv: "224d07175e44f5222ba6103add0438a2"
server_key_version: "encryptionkey@0"
I20260620 09:16:37.353600 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "56ff42249922481ea6ced5ce70352b86"
format_stamp: "Formatted at 2026-06-20 09:16:37 on dist-test-slave-rq8c"
server_key: "76af7ade1bd12bde068723cea5e76ceb"
server_key_iv: "224d07175e44f5222ba6103add0438a2"
server_key_version: "encryptionkey@0"
I20260620 09:16:37.354811 2469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:16:37.355479 2484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:37.355657 2469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:37.355741 2469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "56ff42249922481ea6ced5ce70352b86"
format_stamp: "Formatted at 2026-06-20 09:16:37 on dist-test-slave-rq8c"
server_key: "76af7ade1bd12bde068723cea5e76ceb"
server_key_iv: "224d07175e44f5222ba6103add0438a2"
server_key_version: "encryptionkey@0"
I20260620 09:16:37.355810 2469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:37.366966 2469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:37.368999 2469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:37.369124 2469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:37.369668 2469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:37.369709 2469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:37.369736 2469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:37.369760 2469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:16:37 dist-test-slave-rq8c krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:37.376147 2469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34227
I20260620 09:16:37.376216 2599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34227 every 8 connection(s)
I20260620 09:16:37.376471 2469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:16:37.378657 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.370035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57803 (local address 127.0.98.62:33227)
0620 09:16:37.370095 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.370098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.370323 (+ 225us) server_negotiation.cc:407] Connection header received
0620 09:16:37.370970 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.370971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.370984 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.371036 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:37.371870 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.372419 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.373567 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.373757 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.373808 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.376166 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:37.376167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:37.376167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:37.376174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:37.377433 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.377789 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.377790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.377790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.377848 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.378063 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.378064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.378064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.378174 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:37.378227 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.378477 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.378553 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:16:37.378801 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.370011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33227 (local address 127.0.98.3:57803)
0620 09:16:37.370188 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.370236 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.370859 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.371124 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.371130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.371466 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:37.371779 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.371786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.372507 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.372508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.373490 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.373492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.373834 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.373834 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.373941 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.374365 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:37.374373 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:37.376080 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:37.377509 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.377514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.377516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.377717 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.377897 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.377898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.377898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.378008 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.378224 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:37.378229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:37.378357 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.378442 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.378588 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260620 09:16:37.379240 2600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33227
I20260620 09:16:37.379315 2600 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:37.379506 2600 heartbeater.cc:511] Master 127.0.98.62:33227 requested a full tablet report, sending...
I20260620 09:16:37.379868 2116 ts_manager.cc:194] Registered new tserver with Master: 56ff42249922481ea6ced5ce70352b86 (127.0.98.3:34227)
I20260620 09:16:37.380236 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57803
I20260620 09:16:37.384786 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.380243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60201 (local address 127.0.98.62:33227)
0620 09:16:37.380278 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.380280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.380477 (+ 197us) server_negotiation.cc:407] Connection header received
0620 09:16:37.380533 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.380534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.380543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.380604 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:37.380864 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.381351 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.382012 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.382180 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.382221 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.382878 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:37.382879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:37.382880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:37.382885 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:37.383908 (+ 1023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.384253 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.384254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.384254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.384302 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.384451 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.384451 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.384451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.384536 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:37.384578 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.384679 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.384727 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:16:37.384814 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.380190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33227 (local address 127.0.98.3:60201)
0620 09:16:37.380428 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.380438 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.380488 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.380607 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.380608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.380752 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:37.380804 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.380806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.381426 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.381427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.381956 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.381957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.382225 (+ 268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.382226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.382267 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.382579 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:37.382583 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:37.382819 (+ 236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:37.383995 (+ 1176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.383996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.383996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.384190 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.384342 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.384343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.384343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.384405 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.384572 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:37.384572 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:37.384605 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.384661 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.384740 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":80,"threads_started":1}
I20260620 09:16:37.385923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 2469
I20260620 09:16:37.386031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:16:37.386237 392 external_mini_cluster.cc:1428] Setting key 5c8550f431fb01f42cad09e48fcd46c1
I20260620 09:16:37.387217 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:16:37 dist-test-slave-rq8c krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:16:37 dist-test-slave-rq8c krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:37.407088 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.401283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.0.98.62:33227)
0620 09:16:37.401337 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.401339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.401346 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:37.401386 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.401387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.401395 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.401442 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:37.401897 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.402370 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.403046 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.403228 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.403273 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.405218 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:37.405219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:37.405219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:37.405226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:37.406277 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.406575 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.406576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.406576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.406620 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.406744 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.406744 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.406744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.406832 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:37.406898 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.406966 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.407018 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:16:37.408560 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:37.410010 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:37.415958 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.411825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40901 (local address 127.0.0.1:51414)
0620 09:16:37.412084 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.412110 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.412161 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.412649 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.412650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.412651 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:37.412685 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.412687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.413875 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.413876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.414709 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.414711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.415408 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.415409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.415595 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.415727 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.415820 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.415880 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":56,"threads_started":1}
I20260620 09:16:37.416168 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.411819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.0.1:34564)
0620 09:16:37.412060 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.412104 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.412153 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.412540 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.412541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.412543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:37.412602 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.412604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.413505 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.413506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.415004 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.415005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.415964 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.415965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.416028 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.416032 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.416059 (+ 27us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.416100 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":53,"threads_started":1}
I20260620 09:16:37.416245 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.411913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.0.98.1:40901)
0620 09:16:37.412444 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.412449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.412459 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:16:37.412492 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.412494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.412508 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.412583 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:37.412752 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.413781 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.414784 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.415338 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.415400 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.415552 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.416102 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.416148 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":91,"threads_started":1}
I20260620 09:16:37.416371 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.411830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34227 (local address 127.0.0.1:45242)
0620 09:16:37.412240 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.412248 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.412290 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.412469 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.412472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.412474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:37.412534 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.412536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.413829 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.413830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.414982 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.414984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.415947 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.415947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.416012 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.416016 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.416146 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.416185 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":27,"thread_start_us":47,"threads_started":1}
I20260620 09:16:37.416541 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.412259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.0.98.2:45001)
0620 09:16:37.412410 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.412414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.412422 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:37.412454 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.412455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.412468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.412531 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:37.412739 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.413432 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.415078 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.415882 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.416314 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.416384 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.416409 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.416447 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260620 09:16:37.416671 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.412070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.0.98.3:34227)
0620 09:16:37.412229 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.412233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.412284 (+ 51us) server_negotiation.cc:407] Connection header received
0620 09:16:37.412357 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.412359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.412375 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.412450 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:37.412620 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.413749 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.415079 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.415866 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.415946 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.416008 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.416536 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.416580 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260620 09:16:37.416923 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 508216c690d847c9bab617539023506a (DEFAULT_TABLE table=test-table [id=0d6eb6fc95ef4ef7b4679d440ad1af43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:37.417188 2519 tablet_service.cc:1467] Processing CreateTablet for tablet 508216c690d847c9bab617539023506a (DEFAULT_TABLE table=test-table [id=0d6eb6fc95ef4ef7b4679d440ad1af43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:37.417300 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508216c690d847c9bab617539023506a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:37.417490 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508216c690d847c9bab617539023506a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:37.420120 2621 tablet_bootstrap.cc:492] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e: Bootstrap starting.
I20260620 09:16:37.420060 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 508216c690d847c9bab617539023506a (DEFAULT_TABLE table=test-table [id=0d6eb6fc95ef4ef7b4679d440ad1af43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:37.420357 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508216c690d847c9bab617539023506a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:37.420699 2621 tablet_bootstrap.cc:654] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:37.421010 2621 log.cc:824] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:37.421622 2621 tablet_bootstrap.cc:492] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e: No bootstrap required, opened a new log
I20260620 09:16:37.421674 2621 ts_tablet_manager.cc:1397] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:37.423085 2623 tablet_bootstrap.cc:492] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248: Bootstrap starting.
I20260620 09:16:37.423230 2620 tablet_bootstrap.cc:492] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86: Bootstrap starting.
I20260620 09:16:37.423239 2621 raft_consensus.cc:348] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.423411 2621 raft_consensus.cc:374] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:37.423449 2621 raft_consensus.cc:729] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c5b69d877541229586cb7f981a6d1e, State: Initialized, Role: FOLLOWER
I20260620 09:16:37.423570 2621 consensus_queue.cc:260] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.423712 2623 tablet_bootstrap.cc:654] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:37.423880 2621 ts_tablet_manager.cc:1428] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:16:37.423905 2320 heartbeater.cc:503] Master 127.0.98.62:33227 was elected leader, sending a full tablet report...
I20260620 09:16:37.424011 2623 log.cc:824] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:37.424183 2620 tablet_bootstrap.cc:654] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:37.424460 2620 log.cc:824] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:37.424785 2623 tablet_bootstrap.cc:492] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248: No bootstrap required, opened a new log
I20260620 09:16:37.424889 2623 ts_tablet_manager.cc:1397] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 09:16:37.425143 2620 tablet_bootstrap.cc:492] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86: No bootstrap required, opened a new log
I20260620 09:16:37.425218 2620 ts_tablet_manager.cc:1397] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:16:37.426754 2623 raft_consensus.cc:348] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.426915 2623 raft_consensus.cc:374] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:37.426954 2623 raft_consensus.cc:729] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cf23263eda4efaaac4be509e733248, State: Initialized, Role: FOLLOWER
I20260620 09:16:37.427079 2623 consensus_queue.cc:260] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.427081 2620 raft_consensus.cc:348] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.427239 2620 raft_consensus.cc:374] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:37.427281 2620 raft_consensus.cc:729] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ff42249922481ea6ced5ce70352b86, State: Initialized, Role: FOLLOWER
I20260620 09:16:37.427336 2623 ts_tablet_manager.cc:1428] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 09:16:37.427379 2460 heartbeater.cc:503] Master 127.0.98.62:33227 was elected leader, sending a full tablet report...
I20260620 09:16:37.427407 2620 consensus_queue.cc:260] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.427670 2620 ts_tablet_manager.cc:1428] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:16:37.427802 2600 heartbeater.cc:503] Master 127.0.98.62:33227 was elected leader, sending a full tablet report...
W20260620 09:16:37.484053 2461 tablet.cc:2369] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:16:37.541687 2627 raft_consensus.cc:482] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:37.541806 2627 raft_consensus.cc:504] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.542124 2627 leader_election.cc:288] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901), 56ff42249922481ea6ced5ce70352b86 (127.0.98.3:34227)
I20260620 09:16:37.545894 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.542446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33725 (local address 127.0.98.3:34227)
0620 09:16:37.542483 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.542485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.542492 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:37.542518 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.542518 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.542526 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.542577 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:37.542743 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.543479 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.544992 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.545542 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.545612 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.545622 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.545786 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.545822 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:16:37.545894 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.542405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52819 (local address 127.0.98.1:40901)
0620 09:16:37.542455 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.542457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.542631 (+ 174us) server_negotiation.cc:407] Connection header received
0620 09:16:37.542658 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.542659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.542669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.542710 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:37.542830 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.543500 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.544996 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.545536 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.545612 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.545622 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.545785 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.545820 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:16:37.545917 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.542324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34227 (local address 127.0.98.2:33725)
0620 09:16:37.542372 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.542384 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.542439 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.542604 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.542605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.542606 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:37.542677 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.542679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.543586 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.543587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.544904 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.544908 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.545622 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.545623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.545717 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.545722 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.545776 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.545822 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":11}
I20260620 09:16:37.545941 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.542330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40901 (local address 127.0.98.2:52819)
0620 09:16:37.542560 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.542568 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.542620 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.542729 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.542730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.542732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:37.542780 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.542781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.543586 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.543587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.544904 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.544908 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.545622 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.545623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.545717 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.545722 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.545776 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.545822 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":159,"thread_start_us":111,"threads_started":1}
I20260620 09:16:37.546255 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508216c690d847c9bab617539023506a" candidate_uuid: "30cf23263eda4efaaac4be509e733248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ff42249922481ea6ced5ce70352b86" is_pre_election: true
I20260620 09:16:37.546298 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508216c690d847c9bab617539023506a" candidate_uuid: "30cf23263eda4efaaac4be509e733248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c5b69d877541229586cb7f981a6d1e" is_pre_election: true
I20260620 09:16:37.546399 2554 raft_consensus.cc:2454] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30cf23263eda4efaaac4be509e733248 in term 0.
I20260620 09:16:37.546445 2274 raft_consensus.cc:2454] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30cf23263eda4efaaac4be509e733248 in term 0.
I20260620 09:16:37.546620 2348 leader_election.cc:302] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30cf23263eda4efaaac4be509e733248, 56ff42249922481ea6ced5ce70352b86; no voters:
I20260620 09:16:37.546751 2627 raft_consensus.cc:2781] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:37.546795 2627 raft_consensus.cc:482] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:37.546814 2627 raft_consensus.cc:3037] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:37.547784 2627 raft_consensus.cc:504] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.547904 2627 leader_election.cc:288] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [CANDIDATE]: Term 1 election: Requested vote from peers f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901), 56ff42249922481ea6ced5ce70352b86 (127.0.98.3:34227)
I20260620 09:16:37.548036 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508216c690d847c9bab617539023506a" candidate_uuid: "30cf23263eda4efaaac4be509e733248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c5b69d877541229586cb7f981a6d1e"
I20260620 09:16:37.548048 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508216c690d847c9bab617539023506a" candidate_uuid: "30cf23263eda4efaaac4be509e733248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ff42249922481ea6ced5ce70352b86"
I20260620 09:16:37.548105 2554 raft_consensus.cc:3037] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:37.548102 2274 raft_consensus.cc:3037] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:37.548733 2554 raft_consensus.cc:2454] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30cf23263eda4efaaac4be509e733248 in term 1.
I20260620 09:16:37.548797 2274 raft_consensus.cc:2454] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30cf23263eda4efaaac4be509e733248 in term 1.
I20260620 09:16:37.548892 2348 leader_election.cc:302] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30cf23263eda4efaaac4be509e733248, 56ff42249922481ea6ced5ce70352b86; no voters:
I20260620 09:16:37.548967 2627 raft_consensus.cc:2781] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:37.549101 2627 raft_consensus.cc:686] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 1 LEADER]: Becoming Leader. State: Replica: 30cf23263eda4efaaac4be509e733248, State: Running, Role: LEADER
I20260620 09:16:37.549183 2627 consensus_queue.cc:237] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.549916 2117 catalog_manager.cc:5495] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30cf23263eda4efaaac4be509e733248 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "30cf23263eda4efaaac4be509e733248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } health_report { overall_health: UNKNOWN } } }
W20260620 09:16:37.584794 2321 tablet.cc:2369] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 09:16:37 dist-test-slave-rq8c krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:37.590778 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.584214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33227 (local address 127.0.0.1:41474)
0620 09:16:37.584357 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.584367 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.584433 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.584568 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.584569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.584871 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:37.584913 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.584915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.585571 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.585572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.586164 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.586166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.586486 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.586486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.586536 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.587010 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:37.587015 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:37.588454 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:37.589821 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.589827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.589829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.590032 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.590201 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:37.590201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:37.590202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:37.590252 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:37.590508 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:37.590512 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:37.590552 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.590600 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.590687 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260620 09:16:37.590833 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.584277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.0.98.62:33227)
0620 09:16:37.584348 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.584350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.584422 (+ 72us) server_negotiation.cc:407] Connection header received
0620 09:16:37.584484 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.584485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.584495 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.584563 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:37.584995 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.585497 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.586235 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.586430 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.586477 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.588554 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:37.588555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:37.588556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:37.588561 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:37.589737 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.590101 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.590102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.590102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.590151 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:37.590324 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:37.590325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:37.590325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:37.590448 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:37.590517 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.590626 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.590735 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":45}
I20260620 09:16:37.592767 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41474:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 09:16:37.592957 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:37.594255 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 69625b4b6c65487bb569f7a6957eb151 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=090bd16b7eb1430dbc7813c46b2f4c37]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:16:37.594291 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 69625b4b6c65487bb569f7a6957eb151 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=090bd16b7eb1430dbc7813c46b2f4c37]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:16:37.594290 2519 tablet_service.cc:1467] Processing CreateTablet for tablet 69625b4b6c65487bb569f7a6957eb151 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=090bd16b7eb1430dbc7813c46b2f4c37]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:16:37.594424 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69625b4b6c65487bb569f7a6957eb151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:37.594424 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69625b4b6c65487bb569f7a6957eb151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:37.594427 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69625b4b6c65487bb569f7a6957eb151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:37.595988 2621 tablet_bootstrap.cc:492] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e: Bootstrap starting.
I20260620 09:16:37.596122 2620 tablet_bootstrap.cc:492] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: Bootstrap starting.
I20260620 09:16:37.596364 2621 tablet_bootstrap.cc:654] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:37.596570 2620 tablet_bootstrap.cc:654] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:37.596604 2623 tablet_bootstrap.cc:492] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248: Bootstrap starting.
I20260620 09:16:37.596925 2623 tablet_bootstrap.cc:654] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:37.596940 2621 tablet_bootstrap.cc:492] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e: No bootstrap required, opened a new log
I20260620 09:16:37.596973 2621 ts_tablet_manager.cc:1397] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:37.597131 2620 tablet_bootstrap.cc:492] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: No bootstrap required, opened a new log
I20260620 09:16:37.597162 2620 ts_tablet_manager.cc:1397] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:37.597127 2621 raft_consensus.cc:348] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.597191 2621 raft_consensus.cc:374] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:37.597209 2621 raft_consensus.cc:729] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c5b69d877541229586cb7f981a6d1e, State: Initialized, Role: FOLLOWER
I20260620 09:16:37.597249 2621 consensus_queue.cc:260] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.597278 2620 raft_consensus.cc:348] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.597340 2620 raft_consensus.cc:374] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:37.597347 2621 ts_tablet_manager.cc:1428] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:37.597357 2620 raft_consensus.cc:729] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ff42249922481ea6ced5ce70352b86, State: Initialized, Role: FOLLOWER
I20260620 09:16:37.597420 2620 consensus_queue.cc:260] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.597456 2624 tablet_replica.cc:439] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } } }
I20260620 09:16:37.597512 2624 tablet_replica.cc:442] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:37.597599 2628 tablet_replica.cc:439] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } } }
I20260620 09:16:37.597671 2628 tablet_replica.cc:442] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:37.597704 2623 tablet_bootstrap.cc:492] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248: No bootstrap required, opened a new log
I20260620 09:16:37.597741 2623 ts_tablet_manager.cc:1397] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:37.597741 2620 ts_tablet_manager.cc:1428] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:37.597870 2623 raft_consensus.cc:348] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.597951 2623 raft_consensus.cc:374] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:37.597965 2623 raft_consensus.cc:729] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cf23263eda4efaaac4be509e733248, State: Initialized, Role: FOLLOWER
I20260620 09:16:37.597996 2623 consensus_queue.cc:260] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.598071 2623 ts_tablet_manager.cc:1428] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:37.598115 2627 tablet_replica.cc:439] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } } }
I20260620 09:16:37.598191 2627 tablet_replica.cc:442] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 09:16:37.627070 2601 tablet.cc:2369] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:16:37.772764 2628 raft_consensus.cc:482] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:37.772856 2628 raft_consensus.cc:504] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.773185 2628 leader_election.cc:288] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901), 30cf23263eda4efaaac4be509e733248 (127.0.98.2:45001)
I20260620 09:16:37.776031 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.773365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40901 (local address 127.0.98.3:43705)
0620 09:16:37.773420 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.773431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.773473 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.773652 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.773654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.773656 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:37.773729 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.773731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.774365 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.774365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.775256 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.775258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.775768 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.775769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.775844 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.775848 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.775915 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.775956 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":15}
I20260620 09:16:37.776038 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.773432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43705 (local address 127.0.98.1:40901)
0620 09:16:37.773523 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.773525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.773533 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:37.773563 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.773563 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.773573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.773630 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:37.773803 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.774300 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.775323 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.775707 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.775757 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.775766 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.775921 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.775959 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 09:16:37.776157 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.773422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.98.3:51943)
0620 09:16:37.773575 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.773583 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.773628 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.773748 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.773749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.773750 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:37.773819 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.773820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.774434 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.774435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.775274 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.775276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.775901 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.775902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.775965 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.775969 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.776047 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.776083 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260620 09:16:37.776168 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.773459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51943 (local address 127.0.98.2:45001)
0620 09:16:37.773512 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.773514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.773635 (+ 121us) server_negotiation.cc:407] Connection header received
0620 09:16:37.773666 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.773667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.773679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.773743 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:37.773875 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.774364 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.775400 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.775808 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.775876 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.775886 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.776061 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.776106 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:16:37.776396 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69625b4b6c65487bb569f7a6957eb151" candidate_uuid: "56ff42249922481ea6ced5ce70352b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c5b69d877541229586cb7f981a6d1e" is_pre_election: true
I20260620 09:16:37.776468 2274 raft_consensus.cc:2454] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56ff42249922481ea6ced5ce70352b86 in term 0.
I20260620 09:16:37.776440 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69625b4b6c65487bb569f7a6957eb151" candidate_uuid: "56ff42249922481ea6ced5ce70352b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cf23263eda4efaaac4be509e733248" is_pre_election: true
I20260620 09:16:37.776558 2414 raft_consensus.cc:2454] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56ff42249922481ea6ced5ce70352b86 in term 0.
I20260620 09:16:37.776611 2487 leader_election.cc:302] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ff42249922481ea6ced5ce70352b86, f8c5b69d877541229586cb7f981a6d1e; no voters:
I20260620 09:16:37.776728 2628 raft_consensus.cc:2781] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:37.776773 2628 raft_consensus.cc:482] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:37.776794 2628 raft_consensus.cc:3037] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:37.777451 2628 raft_consensus.cc:504] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.777565 2628 leader_election.cc:288] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [CANDIDATE]: Term 1 election: Requested vote from peers f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901), 30cf23263eda4efaaac4be509e733248 (127.0.98.2:45001)
I20260620 09:16:37.777709 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69625b4b6c65487bb569f7a6957eb151" candidate_uuid: "56ff42249922481ea6ced5ce70352b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cf23263eda4efaaac4be509e733248"
I20260620 09:16:37.777702 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69625b4b6c65487bb569f7a6957eb151" candidate_uuid: "56ff42249922481ea6ced5ce70352b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c5b69d877541229586cb7f981a6d1e"
I20260620 09:16:37.777773 2414 raft_consensus.cc:3037] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:37.777773 2274 raft_consensus.cc:3037] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:37.778455 2274 raft_consensus.cc:2454] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56ff42249922481ea6ced5ce70352b86 in term 1.
I20260620 09:16:37.778455 2414 raft_consensus.cc:2454] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56ff42249922481ea6ced5ce70352b86 in term 1.
I20260620 09:16:37.778620 2487 leader_election.cc:302] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ff42249922481ea6ced5ce70352b86, f8c5b69d877541229586cb7f981a6d1e; no voters:
I20260620 09:16:37.778708 2628 raft_consensus.cc:2781] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:37.778863 2628 raft_consensus.cc:686] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [term 1 LEADER]: Becoming Leader. State: Replica: 56ff42249922481ea6ced5ce70352b86, State: Running, Role: LEADER
I20260620 09:16:37.778985 2628 consensus_queue.cc:237] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.779438 2642 tablet_replica.cc:439] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: TxnStatusTablet state changed. Reason: New leader 56ff42249922481ea6ced5ce70352b86. Latest consensus state: current_term: 1 leader_uuid: "56ff42249922481ea6ced5ce70352b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } } }
I20260620 09:16:37.779505 2642 tablet_replica.cc:442] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:16:37.779765 2116 catalog_manager.cc:5495] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56ff42249922481ea6ced5ce70352b86 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "56ff42249922481ea6ced5ce70352b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } health_report { overall_health: HEALTHY } } }
I20260620 09:16:37.780037 2644 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 09:16:37.780133 2644 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 09:16:37.825376 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.823064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45252 (local address 127.0.98.3:34227)
0620 09:16:37.823113 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.823115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.823124 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:16:37.823219 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.823220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.823238 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.823287 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:37.823415 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.823925 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.824654 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.824834 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.824897 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.825052 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:37.825148 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:37.825215 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.825274 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.825295 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:16:37.825385 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.823007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34227 (local address 127.0.0.1:45252)
0620 09:16:37.823077 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.823090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.823138 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.823297 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.823297 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.823299 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:16:37.823349 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.823351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.823996 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.823997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.824577 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.824579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.824894 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.824895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.824957 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.824972 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:16:37.825204 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:16:37.825211 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.825267 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.825313 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 09:16:37.826972 2414 raft_consensus.cc:1261] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Refusing update from remote peer 56ff42249922481ea6ced5ce70352b86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:37.826972 2274 raft_consensus.cc:1261] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Refusing update from remote peer 56ff42249922481ea6ced5ce70352b86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:37.827244 2642 consensus_queue.cc:1035] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:37.827368 2628 consensus_queue.cc:1035] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:37.828469 2627 tablet_replica.cc:439] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248: TxnStatusTablet state changed. Reason: New leader 56ff42249922481ea6ced5ce70352b86. Latest consensus state: current_term: 1 leader_uuid: "56ff42249922481ea6ced5ce70352b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } } }
I20260620 09:16:37.828564 2627 tablet_replica.cc:442] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:37.828698 2624 tablet_replica.cc:439] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e: TxnStatusTablet state changed. Reason: New leader 56ff42249922481ea6ced5ce70352b86. Latest consensus state: current_term: 1 leader_uuid: "56ff42249922481ea6ced5ce70352b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } } }
I20260620 09:16:37.828771 2624 tablet_replica.cc:442] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:37.829284 2642 tablet_replica.cc:439] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "56ff42249922481ea6ced5ce70352b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } } }
I20260620 09:16:37.829367 2642 tablet_replica.cc:442] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:16:37.829455 2647 mvcc.cc:205] Tried to move back new op lower bound from 7298854903097057280 to 7298854902903410688. Current Snapshot: MvccSnapshot[applied={T|T < 7298854903097057280}]
I20260620 09:16:37.829648 2648 mvcc.cc:205] Tried to move back new op lower bound from 7298854903097057280 to 7298854902903410688. Current Snapshot: MvccSnapshot[applied={T|T < 7298854903097057280}]
I20260620 09:16:37.830166 2624 tablet_replica.cc:439] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "56ff42249922481ea6ced5ce70352b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } } }
I20260620 09:16:37.830248 2624 tablet_replica.cc:442] T 69625b4b6c65487bb569f7a6957eb151 P f8c5b69d877541229586cb7f981a6d1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:37.831281 2628 tablet_replica.cc:439] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "56ff42249922481ea6ced5ce70352b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } } }
I20260620 09:16:37.831363 2628 tablet_replica.cc:442] T 69625b4b6c65487bb569f7a6957eb151 P 56ff42249922481ea6ced5ce70352b86: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:16:37.831984 2627 tablet_replica.cc:439] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "56ff42249922481ea6ced5ce70352b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } } }
I20260620 09:16:37.832070 2627 tablet_replica.cc:442] T 69625b4b6c65487bb569f7a6957eb151 P 30cf23263eda4efaaac4be509e733248: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:37.833912 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.831492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.0.98.2:45001)
0620 09:16:37.831531 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.831533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.831539 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:16:37.831567 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.831568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.831584 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.831733 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:37.831869 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.832505 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.833257 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.833449 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.833500 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.833637 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:37.833724 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:37.833786 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.833817 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.833837 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:16:37.838629 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.836121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49283 (local address 127.0.98.3:34227)
0620 09:16:37.836159 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.836160 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.836234 (+ 74us) server_negotiation.cc:407] Connection header received
0620 09:16:37.836300 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.836301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.836310 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.836349 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:37.836480 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.836936 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.838016 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.838199 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.838258 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.838443 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:37.838466 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:37.838497 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.838556 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.838583 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:16:37.838686 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.836042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34227 (local address 127.0.98.2:49283)
0620 09:16:37.836176 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.836187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.836247 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.836355 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.836356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.836357 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:16:37.836414 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.836416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.837007 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.837008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.837928 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.837931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.838275 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.838276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.838340 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.838354 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:16:37.838507 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:16:37.838509 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.838554 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.838606 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":60,"threads_started":1}
I20260620 09:16:37.840747 2274 raft_consensus.cc:1261] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Refusing update from remote peer 30cf23263eda4efaaac4be509e733248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:37.840763 2554 raft_consensus.cc:1261] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Refusing update from remote peer 30cf23263eda4efaaac4be509e733248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:37.841014 2630 consensus_queue.cc:1035] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:37.841130 2627 consensus_queue.cc:1035] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:37.849289 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.846730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60733 (local address 127.0.98.2:45001)
0620 09:16:37.846765 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.846767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.846793 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:16:37.846845 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.846846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.846854 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.846900 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:37.847029 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.847486 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.848680 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.848884 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.848938 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.849078 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:37.849112 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:37.849155 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.849204 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.849221 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 09:16:37.849303 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.846690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.98.3:60733)
0620 09:16:37.846747 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.846758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.846808 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.846920 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.846921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.846922 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:16:37.846976 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.846978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.847552 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.847553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.848626 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.848628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.848945 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.848946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.849006 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.849019 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:16:37.849163 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:16:37.849165 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.849203 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.849244 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 09:16:37.858206 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:37.858330 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:37.859030 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4: Sending DeleteTablet for 3 replicas of tablet 508216c690d847c9bab617539023506a
I20260620 09:16:37.859329 2519 tablet_service.cc:1514] Processing DeleteTablet for tablet 508216c690d847c9bab617539023506a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45242
I20260620 09:16:37.859382 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet 508216c690d847c9bab617539023506a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51414
I20260620 09:16:37.859539 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 508216c690d847c9bab617539023506a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34564
I20260620 09:16:37.859905 2667 tablet_replica.cc:330] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e: stopping tablet replica
I20260620 09:16:37.860040 2667 raft_consensus.cc:2229] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:37.860034 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 09:16:37.860114 2667 raft_consensus.cc:2258] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 09:16:37.860164 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:37.861356 2254 tablet_service.cc:1467] Processing CreateTablet for tablet ce3e14d52f384d2f9d6f6ca508fc3ca7 (DEFAULT_TABLE table=test-table [id=15b6eec5de5245fe9b1b560bc10347e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:37.861364 2519 tablet_service.cc:1467] Processing CreateTablet for tablet ce3e14d52f384d2f9d6f6ca508fc3ca7 (DEFAULT_TABLE table=test-table [id=15b6eec5de5245fe9b1b560bc10347e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:37.861478 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3e14d52f384d2f9d6f6ca508fc3ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:37.861478 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3e14d52f384d2f9d6f6ca508fc3ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:37.862736 2620 tablet_bootstrap.cc:492] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86: Bootstrap starting.
I20260620 09:16:37.862736 2621 tablet_bootstrap.cc:492] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e: Bootstrap starting.
I20260620 09:16:37.863106 2621 tablet_bootstrap.cc:654] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:37.863106 2620 tablet_bootstrap.cc:654] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:37.863570 2394 tablet_service.cc:1467] Processing CreateTablet for tablet ce3e14d52f384d2f9d6f6ca508fc3ca7 (DEFAULT_TABLE table=test-table [id=15b6eec5de5245fe9b1b560bc10347e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:37.863691 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3e14d52f384d2f9d6f6ca508fc3ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:37.864679 2623 tablet_bootstrap.cc:492] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248: Bootstrap starting.
I20260620 09:16:37.865072 2623 tablet_bootstrap.cc:654] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:37.865301 2667 ts_tablet_manager.cc:1905] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:37.866739 2620 tablet_bootstrap.cc:492] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86: No bootstrap required, opened a new log
I20260620 09:16:37.866787 2620 ts_tablet_manager.cc:1397] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 09:16:37.866956 2620 raft_consensus.cc:348] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.867027 2620 raft_consensus.cc:374] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:37.867024 2667 ts_tablet_manager.cc:1918] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:16:37.867045 2620 raft_consensus.cc:729] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ff42249922481ea6ced5ce70352b86, State: Initialized, Role: FOLLOWER
I20260620 09:16:37.867103 2667 log.cc:1196] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/wal/wals/508216c690d847c9bab617539023506a
I20260620 09:16:37.867091 2620 consensus_queue.cc:260] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.867177 2620 ts_tablet_manager.cc:1428] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:37.867339 2667 ts_tablet_manager.cc:1939] T 508216c690d847c9bab617539023506a P f8c5b69d877541229586cb7f981a6d1e: Deleting consensus metadata
I20260620 09:16:37.867703 2668 tablet_replica.cc:330] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248: stopping tablet replica
I20260620 09:16:37.867703 2666 tablet_replica.cc:330] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86: stopping tablet replica
I20260620 09:16:37.867820 2666 raft_consensus.cc:2229] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:37.867820 2668 raft_consensus.cc:2229] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:37.867877 2666 raft_consensus.cc:2258] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:37.867918 2668 raft_consensus.cc:2258] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:37.868001 2103 catalog_manager.cc:4841] TS f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901): tablet 508216c690d847c9bab617539023506a (table test-table [id=0d6eb6fc95ef4ef7b4679d440ad1af43]) successfully deleted
I20260620 09:16:37.868274 2666 ts_tablet_manager.cc:1905] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:37.868309 2621 tablet_bootstrap.cc:492] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e: No bootstrap required, opened a new log
I20260620 09:16:37.868324 2668 ts_tablet_manager.cc:1905] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:37.868340 2621 ts_tablet_manager.cc:1397] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 09:16:37.868475 2621 raft_consensus.cc:348] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.868539 2621 raft_consensus.cc:374] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:37.868556 2621 raft_consensus.cc:729] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c5b69d877541229586cb7f981a6d1e, State: Initialized, Role: FOLLOWER
I20260620 09:16:37.868600 2621 consensus_queue.cc:260] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.868677 2621 ts_tablet_manager.cc:1428] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:37.869916 2666 ts_tablet_manager.cc:1918] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:16:37.869916 2668 ts_tablet_manager.cc:1918] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:16:37.870002 2668 log.cc:1196] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/508216c690d847c9bab617539023506a
I20260620 09:16:37.870038 2666 log.cc:1196] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/508216c690d847c9bab617539023506a
I20260620 09:16:37.870293 2668 ts_tablet_manager.cc:1939] T 508216c690d847c9bab617539023506a P 30cf23263eda4efaaac4be509e733248: Deleting consensus metadata
I20260620 09:16:37.870309 2666 ts_tablet_manager.cc:1939] T 508216c690d847c9bab617539023506a P 56ff42249922481ea6ced5ce70352b86: Deleting consensus metadata
I20260620 09:16:37.870649 2102 catalog_manager.cc:4841] TS 30cf23263eda4efaaac4be509e733248 (127.0.98.2:45001): tablet 508216c690d847c9bab617539023506a (table test-table [id=0d6eb6fc95ef4ef7b4679d440ad1af43]) successfully deleted
I20260620 09:16:37.870711 2104 catalog_manager.cc:4841] TS 56ff42249922481ea6ced5ce70352b86 (127.0.98.3:34227): tablet 508216c690d847c9bab617539023506a (table test-table [id=0d6eb6fc95ef4ef7b4679d440ad1af43]) successfully deleted
I20260620 09:16:37.872048 2623 tablet_bootstrap.cc:492] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248: No bootstrap required, opened a new log
I20260620 09:16:37.872089 2623 ts_tablet_manager.cc:1397] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260620 09:16:37.872236 2623 raft_consensus.cc:348] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.872290 2623 raft_consensus.cc:374] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:37.872308 2623 raft_consensus.cc:729] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cf23263eda4efaaac4be509e733248, State: Initialized, Role: FOLLOWER
I20260620 09:16:37.872344 2623 consensus_queue.cc:260] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.872423 2623 ts_tablet_manager.cc:1428] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:37.885975 2624 raft_consensus.cc:482] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:37.886062 2624 raft_consensus.cc:504] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.886376 2624 leader_election.cc:288] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30cf23263eda4efaaac4be509e733248 (127.0.98.2:45001), 56ff42249922481ea6ced5ce70352b86 (127.0.98.3:34227)
I20260620 09:16:37.890480 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.886723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53277 (local address 127.0.98.2:45001)
0620 09:16:37.886762 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.886764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.887091 (+ 327us) server_negotiation.cc:407] Connection header received
0620 09:16:37.887173 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.887174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.887182 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.887208 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:37.887373 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.888078 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.889579 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.890120 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.890159 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.890168 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.890380 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.890413 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:16:37.890480 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.886788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37755 (local address 127.0.98.3:34227)
0620 09:16:37.886825 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.886827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.886959 (+ 132us) server_negotiation.cc:407] Connection header received
0620 09:16:37.887010 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.887011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.887020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.887068 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:37.887256 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.888030 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.889579 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.890116 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.890159 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.890168 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.890392 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.890423 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:16:37.890691 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.886523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34227 (local address 127.0.98.1:37755)
0620 09:16:37.886893 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.886903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.886963 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.887111 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.887112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.887113 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:37.887182 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.887184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.888136 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.888137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.889478 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.889482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.890227 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.890227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.890302 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.890307 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.890557 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.890600 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":54,"thread_start_us":82,"threads_started":1}
I20260620 09:16:37.890947 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3e14d52f384d2f9d6f6ca508fc3ca7" candidate_uuid: "f8c5b69d877541229586cb7f981a6d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ff42249922481ea6ced5ce70352b86" is_pre_election: true
I20260620 09:16:37.891033 2554 raft_consensus.cc:2454] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c5b69d877541229586cb7f981a6d1e in term 0.
I20260620 09:16:37.891194 2208 leader_election.cc:302] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ff42249922481ea6ced5ce70352b86, f8c5b69d877541229586cb7f981a6d1e; no voters:
I20260620 09:16:37.891306 2624 raft_consensus.cc:2781] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:37.891351 2624 raft_consensus.cc:482] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:37.891368 2624 raft_consensus.cc:3037] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:37.892063 2624 raft_consensus.cc:504] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.892179 2624 leader_election.cc:288] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [CANDIDATE]: Term 1 election: Requested vote from peers 30cf23263eda4efaaac4be509e733248 (127.0.98.2:45001), 56ff42249922481ea6ced5ce70352b86 (127.0.98.3:34227)
I20260620 09:16:37.891301 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.886523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.98.1:53277)
0620 09:16:37.887044 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.887052 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.887111 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.887260 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.887261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.887262 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:37.887309 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.887310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.888153 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.888154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.889479 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.889482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.890225 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.890226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.890303 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.890305 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.891187 (+ 882us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.891232 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":60,"thread_start_us":70,"threads_started":1}
I20260620 09:16:37.892366 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3e14d52f384d2f9d6f6ca508fc3ca7" candidate_uuid: "f8c5b69d877541229586cb7f981a6d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ff42249922481ea6ced5ce70352b86"
I20260620 09:16:37.892439 2554 raft_consensus.cc:3037] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:37.892540 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3e14d52f384d2f9d6f6ca508fc3ca7" candidate_uuid: "f8c5b69d877541229586cb7f981a6d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cf23263eda4efaaac4be509e733248" is_pre_election: true
I20260620 09:16:37.892609 2414 raft_consensus.cc:2454] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c5b69d877541229586cb7f981a6d1e in term 0.
I20260620 09:16:37.893088 2554 raft_consensus.cc:2454] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c5b69d877541229586cb7f981a6d1e in term 1.
I20260620 09:16:37.893188 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3e14d52f384d2f9d6f6ca508fc3ca7" candidate_uuid: "f8c5b69d877541229586cb7f981a6d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cf23263eda4efaaac4be509e733248"
I20260620 09:16:37.893247 2208 leader_election.cc:302] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ff42249922481ea6ced5ce70352b86, f8c5b69d877541229586cb7f981a6d1e; no voters:
I20260620 09:16:37.893265 2413 raft_consensus.cc:3037] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:37.893317 2624 raft_consensus.cc:2781] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:37.893428 2624 raft_consensus.cc:686] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 1 LEADER]: Becoming Leader. State: Replica: f8c5b69d877541229586cb7f981a6d1e, State: Running, Role: LEADER
I20260620 09:16:37.893488 2624 consensus_queue.cc:237] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:37.894073 2413 raft_consensus.cc:2454] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c5b69d877541229586cb7f981a6d1e in term 1.
I20260620 09:16:37.894109 2117 catalog_manager.cc:5495] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c5b69d877541229586cb7f981a6d1e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f8c5b69d877541229586cb7f981a6d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } health_report { overall_health: UNKNOWN } } }
I20260620 09:16:37.904170 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.901923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.0.98.1:40901)
0620 09:16:37.901959 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.901960 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.901968 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:37.902025 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.902026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.902044 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.902103 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:37.902201 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.902667 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.903454 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.903692 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.903747 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.903882 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:37.903986 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:37.904050 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.904084 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.904101 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:16:37.910051 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.907753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53253 (local address 127.0.98.3:34227)
0620 09:16:37.907790 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.907791 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.907914 (+ 123us) server_negotiation.cc:407] Connection header received
0620 09:16:37.907999 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.908000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.908009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.908035 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:37.908215 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.908746 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.909456 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.909645 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.909697 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.909835 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:37.909867 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:37.909912 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.909968 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.909986 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:16:37.910080 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.906254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34227 (local address 127.0.98.1:53253)
0620 09:16:37.907858 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.907868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.907936 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.908086 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.908086 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.908088 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:16:37.908155 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.908156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.908816 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.908817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.909380 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.909381 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.909700 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.909701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.909760 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.909775 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:16:37.909918 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:16:37.909919 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.909965 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.910004 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1570,"thread_start_us":68,"threads_started":1}
I20260620 09:16:37.912212 2554 raft_consensus.cc:1261] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Refusing update from remote peer f8c5b69d877541229586cb7f981a6d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:37.912212 2413 raft_consensus.cc:1261] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Refusing update from remote peer f8c5b69d877541229586cb7f981a6d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:37.912418 2624 consensus_queue.cc:1035] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:37.912545 2624 consensus_queue.cc:1035] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:37.921099 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.918658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39415 (local address 127.0.98.1:40901)
0620 09:16:37.918692 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:37.918694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:37.918700 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:16:37.918718 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:37.918719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:37.918727 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:37.918766 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:37.919084 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.919565 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.920491 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.920684 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.920744 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.920868 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:37.920902 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:37.920956 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:37.921013 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 09:16:37.921032 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:16:37.921118 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:37.918470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40901 (local address 127.0.98.3:39415)
0620 09:16:37.918540 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:37.918550 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:37.918596 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:37.918802 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:37.918803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:37.918932 (+ 129us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:16:37.918976 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.918978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.919656 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.919657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.920397 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:37.920399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:37.920749 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:37.920750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:37.920803 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:37.920807 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:16:37.920957 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:16:37.920958 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 09:16:37.921009 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 09:16:37.921053 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 09:16:37.921612 2648 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 09:16:39.445442 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:39.445536 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:39.446321 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4: Sending DeleteTablet for 3 replicas of tablet ce3e14d52f384d2f9d6f6ca508fc3ca7
I20260620 09:16:39.446571 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3e14d52f384d2f9d6f6ca508fc3ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34564
I20260620 09:16:39.446590 2519 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3e14d52f384d2f9d6f6ca508fc3ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45242
I20260620 09:16:39.446607 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3e14d52f384d2f9d6f6ca508fc3ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51414
I20260620 09:16:39.446762 2707 tablet_replica.cc:330] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86: stopping tablet replica
I20260620 09:16:39.446839 2707 raft_consensus.cc:2229] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:39.446883 2707 raft_consensus.cc:2258] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:39.448390 2707 ts_tablet_manager.cc:1905] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:39.450030 2707 ts_tablet_manager.cc:1918] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:16:39.450089 2707 log.cc:1196] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/ce3e14d52f384d2f9d6f6ca508fc3ca7
I20260620 09:16:39.450273 2707 ts_tablet_manager.cc:1939] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 56ff42249922481ea6ced5ce70352b86: Deleting consensus metadata
I20260620 09:16:39.450724 2104 catalog_manager.cc:4841] TS 56ff42249922481ea6ced5ce70352b86 (127.0.98.3:34227): tablet ce3e14d52f384d2f9d6f6ca508fc3ca7 (table test-table [id=15b6eec5de5245fe9b1b560bc10347e3]) successfully deleted
I20260620 09:16:39.451334 2705 tablet_replica.cc:330] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248: stopping tablet replica
I20260620 09:16:39.451399 2705 raft_consensus.cc:2229] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:39.451444 2705 raft_consensus.cc:2258] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:39.451691 2706 tablet_replica.cc:330] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e: stopping tablet replica
I20260620 09:16:39.451706 2705 ts_tablet_manager.cc:1905] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:39.451750 2706 raft_consensus.cc:2229] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:39.451860 2706 raft_consensus.cc:2258] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:39.452567 2706 ts_tablet_manager.cc:1905] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:39.453156 2705 ts_tablet_manager.cc:1918] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:16:39.453204 2705 log.cc:1196] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/ce3e14d52f384d2f9d6f6ca508fc3ca7
I20260620 09:16:39.453370 2705 ts_tablet_manager.cc:1939] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P 30cf23263eda4efaaac4be509e733248: Deleting consensus metadata
I20260620 09:16:39.453872 2102 catalog_manager.cc:4841] TS 30cf23263eda4efaaac4be509e733248 (127.0.98.2:45001): tablet ce3e14d52f384d2f9d6f6ca508fc3ca7 (table test-table [id=15b6eec5de5245fe9b1b560bc10347e3]) successfully deleted
I20260620 09:16:39.454150 2706 ts_tablet_manager.cc:1918] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:16:39.454195 2706 log.cc:1196] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/wal/wals/ce3e14d52f384d2f9d6f6ca508fc3ca7
I20260620 09:16:39.454355 2706 ts_tablet_manager.cc:1939] T ce3e14d52f384d2f9d6f6ca508fc3ca7 P f8c5b69d877541229586cb7f981a6d1e: Deleting consensus metadata
I20260620 09:16:39.454675 2103 catalog_manager.cc:4841] TS f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901): tablet ce3e14d52f384d2f9d6f6ca508fc3ca7 (table test-table [id=15b6eec5de5245fe9b1b560bc10347e3]) successfully deleted
Jun 20 09:16:39 dist-test-slave-rq8c krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:16:39 dist-test-slave-rq8c krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:39.469358 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:39.462639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.0.98.62:33227)
0620 09:16:39.462783 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:39.462785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:39.462807 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:16:39.462892 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:39.462893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:39.462904 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:39.462966 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:39.463457 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:39.464090 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:39.464838 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:39.465041 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:39.465092 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:39.467167 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:39.467168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:39.467169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:39.467174 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:39.468484 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:39.468761 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:39.468762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:39.468762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:39.468809 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:39.468959 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:39.468960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:39.468960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:39.469049 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:39.469135 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:39.469206 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 09:16:39.469268 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260620 09:16:39.470638 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:39.470785 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:39.472141 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6f8b7021da4594bd79ddc9cac732f4 (DEFAULT_TABLE table=test-table [id=fdb7d4029b36410e9b297f01a4190551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:39.472169 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6f8b7021da4594bd79ddc9cac732f4 (DEFAULT_TABLE table=test-table [id=fdb7d4029b36410e9b297f01a4190551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:39.472277 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6f8b7021da4594bd79ddc9cac732f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:39.472286 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6f8b7021da4594bd79ddc9cac732f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:39.472265 2519 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6f8b7021da4594bd79ddc9cac732f4 (DEFAULT_TABLE table=test-table [id=fdb7d4029b36410e9b297f01a4190551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:39.472368 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6f8b7021da4594bd79ddc9cac732f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:39.473315 2716 tablet_bootstrap.cc:492] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248: Bootstrap starting.
I20260620 09:16:39.473587 2716 tablet_bootstrap.cc:654] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:39.473677 2717 tablet_bootstrap.cc:492] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86: Bootstrap starting.
I20260620 09:16:39.473991 2717 tablet_bootstrap.cc:654] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:39.474174 2716 tablet_bootstrap.cc:492] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248: No bootstrap required, opened a new log
I20260620 09:16:39.474226 2716 ts_tablet_manager.cc:1397] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:16:39.474290 2718 tablet_bootstrap.cc:492] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e: Bootstrap starting.
I20260620 09:16:39.474339 2716 raft_consensus.cc:348] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.474411 2716 raft_consensus.cc:374] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:39.474466 2716 raft_consensus.cc:729] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cf23263eda4efaaac4be509e733248, State: Initialized, Role: FOLLOWER
I20260620 09:16:39.474532 2716 consensus_queue.cc:260] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.474545 2717 tablet_bootstrap.cc:492] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86: No bootstrap required, opened a new log
I20260620 09:16:39.474597 2717 ts_tablet_manager.cc:1397] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:39.474638 2718 tablet_bootstrap.cc:654] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:39.474700 2716 ts_tablet_manager.cc:1428] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:39.474738 2717 raft_consensus.cc:348] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.474812 2717 raft_consensus.cc:374] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:39.474858 2717 raft_consensus.cc:729] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ff42249922481ea6ced5ce70352b86, State: Initialized, Role: FOLLOWER
I20260620 09:16:39.474926 2717 consensus_queue.cc:260] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.475073 2717 ts_tablet_manager.cc:1428] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:39.475335 2718 tablet_bootstrap.cc:492] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e: No bootstrap required, opened a new log
I20260620 09:16:39.475395 2718 ts_tablet_manager.cc:1397] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:39.475522 2718 raft_consensus.cc:348] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.475594 2718 raft_consensus.cc:374] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:39.475661 2718 raft_consensus.cc:729] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c5b69d877541229586cb7f981a6d1e, State: Initialized, Role: FOLLOWER
I20260620 09:16:39.475715 2718 consensus_queue.cc:260] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.475874 2718 ts_tablet_manager.cc:1428] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:39.563941 2721 raft_consensus.cc:482] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:39.564103 2721 raft_consensus.cc:504] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.564267 2721 leader_election.cc:288] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901), 56ff42249922481ea6ced5ce70352b86 (127.0.98.3:34227)
I20260620 09:16:39.564445 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6f8b7021da4594bd79ddc9cac732f4" candidate_uuid: "30cf23263eda4efaaac4be509e733248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ff42249922481ea6ced5ce70352b86" is_pre_election: true
I20260620 09:16:39.564452 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6f8b7021da4594bd79ddc9cac732f4" candidate_uuid: "30cf23263eda4efaaac4be509e733248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c5b69d877541229586cb7f981a6d1e" is_pre_election: true
I20260620 09:16:39.564540 2554 raft_consensus.cc:2454] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30cf23263eda4efaaac4be509e733248 in term 0.
I20260620 09:16:39.564540 2274 raft_consensus.cc:2454] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30cf23263eda4efaaac4be509e733248 in term 0.
I20260620 09:16:39.564683 2347 leader_election.cc:302] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30cf23263eda4efaaac4be509e733248, f8c5b69d877541229586cb7f981a6d1e; no voters:
I20260620 09:16:39.564785 2721 raft_consensus.cc:2781] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:39.564850 2721 raft_consensus.cc:482] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:39.564864 2721 raft_consensus.cc:3037] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:39.565569 2721 raft_consensus.cc:504] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.565681 2721 leader_election.cc:288] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [CANDIDATE]: Term 1 election: Requested vote from peers f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901), 56ff42249922481ea6ced5ce70352b86 (127.0.98.3:34227)
I20260620 09:16:39.565822 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6f8b7021da4594bd79ddc9cac732f4" candidate_uuid: "30cf23263eda4efaaac4be509e733248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ff42249922481ea6ced5ce70352b86"
I20260620 09:16:39.565892 2554 raft_consensus.cc:3037] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:39.566421 2554 raft_consensus.cc:2454] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30cf23263eda4efaaac4be509e733248 in term 1.
I20260620 09:16:39.566524 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6f8b7021da4594bd79ddc9cac732f4" candidate_uuid: "30cf23263eda4efaaac4be509e733248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c5b69d877541229586cb7f981a6d1e"
I20260620 09:16:39.566565 2348 leader_election.cc:302] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30cf23263eda4efaaac4be509e733248, 56ff42249922481ea6ced5ce70352b86; no voters:
I20260620 09:16:39.566603 2274 raft_consensus.cc:3037] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:39.566649 2721 raft_consensus.cc:2781] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:39.566810 2721 raft_consensus.cc:686] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 1 LEADER]: Becoming Leader. State: Replica: 30cf23263eda4efaaac4be509e733248, State: Running, Role: LEADER
I20260620 09:16:39.566862 2721 consensus_queue.cc:237] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.567291 2274 raft_consensus.cc:2454] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30cf23263eda4efaaac4be509e733248 in term 1.
I20260620 09:16:39.567687 2116 catalog_manager.cc:5495] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30cf23263eda4efaaac4be509e733248 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "30cf23263eda4efaaac4be509e733248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } health_report { overall_health: UNKNOWN } } }
I20260620 09:16:39.592747 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:39.590588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.0.98.2:45001)
0620 09:16:39.590707 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:39.590709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:39.590715 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:16:39.590733 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:39.590733 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:39.590741 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:39.590813 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:39.590921 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:39.591399 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:39.592221 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:39.592413 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:39.592460 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:39.592561 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:39.592586 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:39.592615 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:39.592669 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 09:16:39.592685 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260620 09:16:39.594707 2274 raft_consensus.cc:1261] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Refusing update from remote peer 30cf23263eda4efaaac4be509e733248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:39.594717 2554 raft_consensus.cc:1261] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Refusing update from remote peer 30cf23263eda4efaaac4be509e733248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:39.594916 2723 consensus_queue.cc:1035] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:39.595024 2721 consensus_queue.cc:1035] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:39.604736 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:39.604831 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:39.605372 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4: Sending DeleteTablet for 3 replicas of tablet 0c6f8b7021da4594bd79ddc9cac732f4
I20260620 09:16:39.605582 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6f8b7021da4594bd79ddc9cac732f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34564
I20260620 09:16:39.605611 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6f8b7021da4594bd79ddc9cac732f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51414
I20260620 09:16:39.605630 2519 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6f8b7021da4594bd79ddc9cac732f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45242
I20260620 09:16:39.605674 2705 tablet_replica.cc:330] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248: stopping tablet replica
I20260620 09:16:39.605722 2705 raft_consensus.cc:2229] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:39.605782 2705 raft_consensus.cc:2258] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:39.605885 2706 tablet_replica.cc:330] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e: stopping tablet replica
I20260620 09:16:39.605949 2706 raft_consensus.cc:2229] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:39.605984 2706 raft_consensus.cc:2258] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:39.606047 2707 tablet_replica.cc:330] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86: stopping tablet replica
I20260620 09:16:39.606081 2705 ts_tablet_manager.cc:1905] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:39.606098 2707 raft_consensus.cc:2229] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:39.606132 2707 raft_consensus.cc:2258] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:39.606436 2706 ts_tablet_manager.cc:1905] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:39.606449 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:39.606578 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:39.607484 2705 ts_tablet_manager.cc:1918] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:16:39.607539 2705 log.cc:1196] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/0c6f8b7021da4594bd79ddc9cac732f4
I20260620 09:16:39.607755 2705 ts_tablet_manager.cc:1939] T 0c6f8b7021da4594bd79ddc9cac732f4 P 30cf23263eda4efaaac4be509e733248: Deleting consensus metadata
I20260620 09:16:39.607815 2706 ts_tablet_manager.cc:1918] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:16:39.607874 2706 log.cc:1196] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-0/wal/wals/0c6f8b7021da4594bd79ddc9cac732f4
I20260620 09:16:39.608075 2706 ts_tablet_manager.cc:1939] T 0c6f8b7021da4594bd79ddc9cac732f4 P f8c5b69d877541229586cb7f981a6d1e: Deleting consensus metadata
I20260620 09:16:39.608148 2102 catalog_manager.cc:4841] TS 30cf23263eda4efaaac4be509e733248 (127.0.98.2:45001): tablet 0c6f8b7021da4594bd79ddc9cac732f4 (table test-table [id=fdb7d4029b36410e9b297f01a4190551]) successfully deleted
I20260620 09:16:39.608533 2707 ts_tablet_manager.cc:1905] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:39.608582 2103 catalog_manager.cc:4841] TS f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901): tablet 0c6f8b7021da4594bd79ddc9cac732f4 (table test-table [id=fdb7d4029b36410e9b297f01a4190551]) successfully deleted
I20260620 09:16:39.609758 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd15d73d5c749ff981e09ec4bb2fde1 (DEFAULT_TABLE table=test-table [id=0dbb85d392be414da420b119ab047c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:39.609823 2707 ts_tablet_manager.cc:1918] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:16:39.609886 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd15d73d5c749ff981e09ec4bb2fde1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:39.609903 2707 log.cc:1196] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/0c6f8b7021da4594bd79ddc9cac732f4
I20260620 09:16:39.609817 2519 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd15d73d5c749ff981e09ec4bb2fde1 (DEFAULT_TABLE table=test-table [id=0dbb85d392be414da420b119ab047c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:39.610096 2707 ts_tablet_manager.cc:1939] T 0c6f8b7021da4594bd79ddc9cac732f4 P 56ff42249922481ea6ced5ce70352b86: Deleting consensus metadata
I20260620 09:16:39.610118 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd15d73d5c749ff981e09ec4bb2fde1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:39.610373 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd15d73d5c749ff981e09ec4bb2fde1 (DEFAULT_TABLE table=test-table [id=0dbb85d392be414da420b119ab047c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:39.610484 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd15d73d5c749ff981e09ec4bb2fde1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:39.610569 2104 catalog_manager.cc:4841] TS 56ff42249922481ea6ced5ce70352b86 (127.0.98.3:34227): tablet 0c6f8b7021da4594bd79ddc9cac732f4 (table test-table [id=fdb7d4029b36410e9b297f01a4190551]) successfully deleted
I20260620 09:16:39.611392 2716 tablet_bootstrap.cc:492] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248: Bootstrap starting.
I20260620 09:16:39.611557 2717 tablet_bootstrap.cc:492] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86: Bootstrap starting.
I20260620 09:16:39.611704 2716 tablet_bootstrap.cc:654] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:39.611888 2717 tablet_bootstrap.cc:654] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:39.612247 2716 tablet_bootstrap.cc:492] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248: No bootstrap required, opened a new log
I20260620 09:16:39.612274 2718 tablet_bootstrap.cc:492] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e: Bootstrap starting.
I20260620 09:16:39.612289 2716 ts_tablet_manager.cc:1397] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:39.612396 2716 raft_consensus.cc:348] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.612463 2716 raft_consensus.cc:374] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:39.612493 2716 raft_consensus.cc:729] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cf23263eda4efaaac4be509e733248, State: Initialized, Role: FOLLOWER
I20260620 09:16:39.612534 2716 consensus_queue.cc:260] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.612589 2718 tablet_bootstrap.cc:654] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:39.612632 2716 ts_tablet_manager.cc:1428] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:39.612917 2717 tablet_bootstrap.cc:492] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86: No bootstrap required, opened a new log
I20260620 09:16:39.612967 2717 ts_tablet_manager.cc:1397] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:39.613086 2717 raft_consensus.cc:348] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.613160 2717 raft_consensus.cc:374] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:39.613163 2718 tablet_bootstrap.cc:492] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e: No bootstrap required, opened a new log
I20260620 09:16:39.613189 2717 raft_consensus.cc:729] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ff42249922481ea6ced5ce70352b86, State: Initialized, Role: FOLLOWER
I20260620 09:16:39.613202 2718 ts_tablet_manager.cc:1397] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:39.613240 2717 consensus_queue.cc:260] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.613327 2717 ts_tablet_manager.cc:1428] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:39.613304 2718 raft_consensus.cc:348] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.613368 2718 raft_consensus.cc:374] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:39.613399 2718 raft_consensus.cc:729] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c5b69d877541229586cb7f981a6d1e, State: Initialized, Role: FOLLOWER
I20260620 09:16:39.613436 2718 consensus_queue.cc:260] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.613535 2718 ts_tablet_manager.cc:1428] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:39.829699 2691 raft_consensus.cc:482] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:39.829795 2691 raft_consensus.cc:504] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.829970 2691 leader_election.cc:288] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901), 30cf23263eda4efaaac4be509e733248 (127.0.98.2:45001)
I20260620 09:16:39.830176 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd15d73d5c749ff981e09ec4bb2fde1" candidate_uuid: "56ff42249922481ea6ced5ce70352b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c5b69d877541229586cb7f981a6d1e" is_pre_election: true
I20260620 09:16:39.830193 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd15d73d5c749ff981e09ec4bb2fde1" candidate_uuid: "56ff42249922481ea6ced5ce70352b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cf23263eda4efaaac4be509e733248" is_pre_election: true
I20260620 09:16:39.830271 2413 raft_consensus.cc:2454] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56ff42249922481ea6ced5ce70352b86 in term 0.
I20260620 09:16:39.830271 2274 raft_consensus.cc:2454] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56ff42249922481ea6ced5ce70352b86 in term 0.
I20260620 09:16:39.830415 2486 leader_election.cc:302] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30cf23263eda4efaaac4be509e733248, 56ff42249922481ea6ced5ce70352b86; no voters:
I20260620 09:16:39.830505 2691 raft_consensus.cc:2781] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:39.830544 2691 raft_consensus.cc:482] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:39.830559 2691 raft_consensus.cc:3037] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:39.831203 2691 raft_consensus.cc:504] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.831318 2691 leader_election.cc:288] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [CANDIDATE]: Term 1 election: Requested vote from peers f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901), 30cf23263eda4efaaac4be509e733248 (127.0.98.2:45001)
I20260620 09:16:39.831470 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd15d73d5c749ff981e09ec4bb2fde1" candidate_uuid: "56ff42249922481ea6ced5ce70352b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cf23263eda4efaaac4be509e733248"
I20260620 09:16:39.831535 2413 raft_consensus.cc:3037] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:39.831475 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd15d73d5c749ff981e09ec4bb2fde1" candidate_uuid: "56ff42249922481ea6ced5ce70352b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c5b69d877541229586cb7f981a6d1e"
I20260620 09:16:39.831570 2274 raft_consensus.cc:3037] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:39.832159 2413 raft_consensus.cc:2454] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56ff42249922481ea6ced5ce70352b86 in term 1.
I20260620 09:16:39.832295 2486 leader_election.cc:302] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30cf23263eda4efaaac4be509e733248, 56ff42249922481ea6ced5ce70352b86; no voters:
I20260620 09:16:39.832378 2691 raft_consensus.cc:2781] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:39.832419 2691 raft_consensus.cc:686] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 1 LEADER]: Becoming Leader. State: Replica: 56ff42249922481ea6ced5ce70352b86, State: Running, Role: LEADER
I20260620 09:16:39.832407 2274 raft_consensus.cc:2454] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56ff42249922481ea6ced5ce70352b86 in term 1.
I20260620 09:16:39.832475 2691 consensus_queue.cc:237] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } }
I20260620 09:16:39.833052 2117 catalog_manager.cc:5495] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56ff42249922481ea6ced5ce70352b86 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "56ff42249922481ea6ced5ce70352b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56ff42249922481ea6ced5ce70352b86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34227 } health_report { overall_health: HEALTHY } } }
I20260620 09:16:39.840960 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:39.838251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.0.98.3:34227)
0620 09:16:39.838406 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:39.838408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:39.838428 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:16:39.838450 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:39.838450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:39.838462 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:39.838529 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:39.838669 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:39.839474 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:39.840238 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:39.840560 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:39.840627 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:39.840698 (+ 71us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:39.840733 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:39.840779 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:39.840860 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 09:16:39.840879 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260620 09:16:39.845384 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:39.842980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55403 (local address 127.0.98.3:34227)
0620 09:16:39.843014 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:39.843015 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:39.843079 (+ 64us) server_negotiation.cc:407] Connection header received
0620 09:16:39.843146 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:39.843147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:39.843156 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:39.843206 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:39.843306 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:39.843801 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:39.844834 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:39.845013 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:39.845073 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:39.845203 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:39.845227 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:39.845260 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:39.845311 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 09:16:39.845338 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":17}
I20260620 09:16:39.845487 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:39.842868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34227 (local address 127.0.98.3:55403)
0620 09:16:39.843032 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:39.843041 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:39.843091 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:39.843211 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:39.843211 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:39.843213 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:16:39.843254 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:39.843255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:39.843864 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:39.843865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:39.844747 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:39.844749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:39.845082 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:39.845083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:39.845137 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:39.845141 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:16:39.845262 (+ 121us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:16:39.845263 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 09:16:39.845309 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 09:16:39.845371 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260620 09:16:39.847249 2274 raft_consensus.cc:1261] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Refusing update from remote peer 56ff42249922481ea6ced5ce70352b86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:39.847270 2413 raft_consensus.cc:1261] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Refusing update from remote peer 56ff42249922481ea6ced5ce70352b86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:39.847460 2686 consensus_queue.cc:1035] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30cf23263eda4efaaac4be509e733248" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:39.847551 2691 consensus_queue.cc:1035] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c5b69d877541229586cb7f981a6d1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:16:39.851728 2684 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 09:16:41.368578 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:41.368682 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:41.369339 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04caaa4ce9d7457c887cb18ba3ac60d4: Sending DeleteTablet for 3 replicas of tablet 2cd15d73d5c749ff981e09ec4bb2fde1
I20260620 09:16:41.369554 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd15d73d5c749ff981e09ec4bb2fde1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51414
I20260620 09:16:41.369575 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd15d73d5c749ff981e09ec4bb2fde1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34564
I20260620 09:16:41.369726 2519 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd15d73d5c749ff981e09ec4bb2fde1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45242
I20260620 09:16:41.369753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 2189
I20260620 09:16:41.369760 2761 tablet_replica.cc:330] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e: stopping tablet replica
I20260620 09:16:41.369817 2761 raft_consensus.cc:2229] T 2cd15d73d5c749ff981e09ec4bb2fde1 P f8c5b69d877541229586cb7f981a6d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:41.370955 2762 tablet_replica.cc:330] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248: stopping tablet replica
I20260620 09:16:41.371021 2762 raft_consensus.cc:2229] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:41.371059 2762 raft_consensus.cc:2258] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:41.371325 2762 ts_tablet_manager.cc:1905] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:41.371687 2763 tablet_replica.cc:330] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86: stopping tablet replica
I20260620 09:16:41.371742 2763 raft_consensus.cc:2229] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:41.371850 2763 raft_consensus.cc:2258] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:41.372169 2763 ts_tablet_manager.cc:1905] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:41.373153 2762 ts_tablet_manager.cc:1918] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:16:41.373214 2762 log.cc:1196] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/2cd15d73d5c749ff981e09ec4bb2fde1
I20260620 09:16:41.373411 2762 ts_tablet_manager.cc:1939] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 30cf23263eda4efaaac4be509e733248: Deleting consensus metadata
I20260620 09:16:41.373760 2763 ts_tablet_manager.cc:1918] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:16:41.373838 2763 log.cc:1196] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/2cd15d73d5c749ff981e09ec4bb2fde1
I20260620 09:16:41.373829 2102 catalog_manager.cc:4841] TS 30cf23263eda4efaaac4be509e733248 (127.0.98.2:45001): tablet 2cd15d73d5c749ff981e09ec4bb2fde1 (table test-table [id=0dbb85d392be414da420b119ab047c97]) successfully deleted
I20260620 09:16:41.374055 2763 ts_tablet_manager.cc:1939] T 2cd15d73d5c749ff981e09ec4bb2fde1 P 56ff42249922481ea6ced5ce70352b86: Deleting consensus metadata
I20260620 09:16:41.374321 2104 catalog_manager.cc:4841] TS 56ff42249922481ea6ced5ce70352b86 (127.0.98.3:34227): tablet 2cd15d73d5c749ff981e09ec4bb2fde1 (table test-table [id=0dbb85d392be414da420b119ab047c97]) successfully deleted
W20260620 09:16:41.375486 2103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40901 (error 108)
I20260620 09:16:41.375720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 2329
I20260620 09:16:41.381597 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:41.375838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40901 (local address 127.0.0.1:51428)
0620 09:16:41.381497 (+ 5659us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:41.381557 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5633,"thread_start_us":53,"threads_started":1}
W20260620 09:16:41.381711 2103 catalog_manager.cc:4584] TS f8c5b69d877541229586cb7f981a6d1e (127.0.98.1:40901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cd15d73d5c749ff981e09ec4bb2fde1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40901: connect: Connection refused (error 111)
I20260620 09:16:41.381747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 2469
I20260620 09:16:41.386498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 2085
2026-06-20T09:16:41Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5668 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:16:41 dist-test-slave-rq8c krb5kdc[2768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:16:41 dist-test-slave-rq8c krb5kdc[2768](info): set up 2 sockets
Jun 20 09:16:41 dist-test-slave-rq8c krb5kdc[2768](info): commencing operation
krb5kdc: starting...
W20260620 09:16:42.335359 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:16:42 dist-test-slave-rq8c krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:16:42.440289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:42.517472 2781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:42.517673 2781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:42.517700 2781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:42.518838 2781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:42.518879 2781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:42.518898 2781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:42.518913 2781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:42.518929 2781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:42.520411 2781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:42.520645 2781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:42.520843 2781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:42.525276 2787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:42.525327 2786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:42.525484 2781 server_base.cc:1034] running on GCE node
W20260620 09:16:42.525333 2789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:42.525776 2781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:42.526059 2781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:42.527189 2781 hybrid_clock.cc:648] HybridClock initialized: now 1781947002527179 us; error 27 us; skew 500 ppm
Jun 20 09:16:42 dist-test-slave-rq8c krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:42.529659 2781 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:16:42.530045 2781 webserver.cc:460] Webserver started at http://127.0.98.62:36939/ using document root <none> and password file <none>
I20260620 09:16:42.530247 2781 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:42.530292 2781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:42.530413 2781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:42.531409 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "b72ba142a7a549b3874d370ac8b9ffca"
format_stamp: "Formatted at 2026-06-20 09:16:42 on dist-test-slave-rq8c"
server_key: "7e40d308abd3cf6946d5acbeba6cf64f"
server_key_iv: "a10474c72bb053c9eaf65c5ed6850784"
server_key_version: "encryptionkey@0"
I20260620 09:16:42.531760 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "b72ba142a7a549b3874d370ac8b9ffca"
format_stamp: "Formatted at 2026-06-20 09:16:42 on dist-test-slave-rq8c"
server_key: "7e40d308abd3cf6946d5acbeba6cf64f"
server_key_iv: "a10474c72bb053c9eaf65c5ed6850784"
server_key_version: "encryptionkey@0"
I20260620 09:16:42.532972 2781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:42.533632 2796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:42.533779 2781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:42.533879 2781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "b72ba142a7a549b3874d370ac8b9ffca"
format_stamp: "Formatted at 2026-06-20 09:16:42 on dist-test-slave-rq8c"
server_key: "7e40d308abd3cf6946d5acbeba6cf64f"
server_key_iv: "a10474c72bb053c9eaf65c5ed6850784"
server_key_version: "encryptionkey@0"
I20260620 09:16:42.533939 2781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:42.542877 2781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:42.544540 2781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:42.544643 2781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:42.548096 2781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42111
I20260620 09:16:42.548132 2848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42111 every 8 connection(s)
I20260620 09:16:42.548560 2781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:42.549170 2849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:42.551250 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca: Bootstrap starting.
I20260620 09:16:42.551811 2849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:42.552057 2849 log.cc:824] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:42.552649 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca: No bootstrap required, opened a new log
I20260620 09:16:42.554171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 2781
I20260620 09:16:42.554121 2849 raft_consensus.cc:348] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72ba142a7a549b3874d370ac8b9ffca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42111 } }
I20260620 09:16:42.554248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:42.554262 2849 raft_consensus.cc:374] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:42.554306 2849 raft_consensus.cc:729] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72ba142a7a549b3874d370ac8b9ffca, State: Initialized, Role: FOLLOWER
I20260620 09:16:42.554399 392 external_mini_cluster.cc:1428] Setting key 546af92281f9e5436cff86949046dc65
I20260620 09:16:42.554443 2849 consensus_queue.cc:260] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72ba142a7a549b3874d370ac8b9ffca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42111 } }
I20260620 09:16:42.554517 2849 raft_consensus.cc:388] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:42.554551 2849 raft_consensus.cc:482] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:42.554589 2849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:42.555735 2849 raft_consensus.cc:504] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72ba142a7a549b3874d370ac8b9ffca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42111 } }
I20260620 09:16:42.555843 2849 leader_election.cc:302] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b72ba142a7a549b3874d370ac8b9ffca; no voters:
I20260620 09:16:42.556005 2849 leader_election.cc:288] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:42.556046 2854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:42.556198 2854 raft_consensus.cc:686] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [term 1 LEADER]: Becoming Leader. State: Replica: b72ba142a7a549b3874d370ac8b9ffca, State: Running, Role: LEADER
I20260620 09:16:42.556278 2849 sys_catalog.cc:564] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:42.556330 2854 consensus_queue.cc:237] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72ba142a7a549b3874d370ac8b9ffca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42111 } }
I20260620 09:16:42.556677 2854 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b72ba142a7a549b3874d370ac8b9ffca. Latest consensus state: current_term: 1 leader_uuid: "b72ba142a7a549b3874d370ac8b9ffca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72ba142a7a549b3874d370ac8b9ffca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42111 } } }
I20260620 09:16:42.556911 2854 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:42.556842 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b72ba142a7a549b3874d370ac8b9ffca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72ba142a7a549b3874d370ac8b9ffca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42111 } } }
I20260620 09:16:42.557255 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca [sys.catalog]: This master's current role is: LEADER
W20260620 09:16:42.557948 2870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:16:42.558000 2870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:16:42.558038 2863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:42.558174 2863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:42.559456 2863 catalog_manager.cc:1347] Generated new cluster ID: 892b1446fa144046860f61ffd9ff5e95
I20260620 09:16:42.559501 2863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:16:42 dist-test-slave-rq8c krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:42.569080 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:42.554816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.0.98.62:42111)
0620 09:16:42.554974 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:42.554982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:42.555007 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:16:42.555104 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:42.555109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:42.555158 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:42.555275 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:42.556022 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.557358 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.558400 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.559005 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.559104 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:42.565637 (+ 6533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:42.565641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:42.565641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:42.565654 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:42.567523 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:42.567897 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:42.567900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:42.567900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:42.568004 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:42.568178 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:42.568179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:42.568179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:42.568484 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:42.568616 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:42.568715 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 09:16:42.568853 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260620 09:16:42.578090 2863 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:42.578581 2863 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:42.583871 2863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b72ba142a7a549b3874d370ac8b9ffca: Generated new TSK 0
I20260620 09:16:42.584033 2863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:16:42.612313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42111
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:42.690801 2875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:42.690992 2875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:42.691013 2875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:42.692159 2875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:42.692198 2875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:42.692226 2875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:42.693703 2875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:42.693917 2875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:42.694123 2875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:42.698786 2883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:42.698788 2880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:42.698827 2875 server_base.cc:1034] running on GCE node
W20260620 09:16:42.698786 2881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:42.699208 2875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:42.699422 2875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:42.700565 2875 hybrid_clock.cc:648] HybridClock initialized: now 1781947002700555 us; error 29 us; skew 500 ppm
Jun 20 09:16:42 dist-test-slave-rq8c krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:42.702876 2875 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:16:42.703271 2875 webserver.cc:460] Webserver started at http://127.0.98.1:36389/ using document root <none> and password file <none>
I20260620 09:16:42.703433 2875 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:42.703470 2875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:42.703558 2875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:42.704537 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "e141deaec6b14b0382f4f2f6d59d6eb0"
format_stamp: "Formatted at 2026-06-20 09:16:42 on dist-test-slave-rq8c"
server_key: "84a48faa0f84e96ce176eab939afa652"
server_key_iv: "09601fb7d4e02ccd7e36eafa80e04d03"
server_key_version: "encryptionkey@0"
I20260620 09:16:42.704824 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e141deaec6b14b0382f4f2f6d59d6eb0"
format_stamp: "Formatted at 2026-06-20 09:16:42 on dist-test-slave-rq8c"
server_key: "84a48faa0f84e96ce176eab939afa652"
server_key_iv: "09601fb7d4e02ccd7e36eafa80e04d03"
server_key_version: "encryptionkey@0"
I20260620 09:16:42.706025 2875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:16:42.706627 2890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:42.706754 2875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:42.706832 2875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "e141deaec6b14b0382f4f2f6d59d6eb0"
format_stamp: "Formatted at 2026-06-20 09:16:42 on dist-test-slave-rq8c"
server_key: "84a48faa0f84e96ce176eab939afa652"
server_key_iv: "09601fb7d4e02ccd7e36eafa80e04d03"
server_key_version: "encryptionkey@0"
I20260620 09:16:42.706889 2875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:42.716756 2875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:42.718724 2875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:42.718843 2875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:42.719048 2875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:42.719327 2875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:42.719360 2875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:42.719388 2875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:42.719412 2875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:42.724813 2875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38389
I20260620 09:16:42.724874 3003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38389 every 8 connection(s)
I20260620 09:16:42.725148 2875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:16:42.726305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 2875
I20260620 09:16:42.726379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:16:42.726553 392 external_mini_cluster.cc:1428] Setting key ae8ea58025aec346cb5cc09313858c78
Jun 20 09:16:42 dist-test-slave-rq8c krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:16:42.735284 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:42.725848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35083 (local address 127.0.98.62:42111)
0620 09:16:42.725905 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:42.725907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:42.726105 (+ 198us) server_negotiation.cc:407] Connection header received
0620 09:16:42.727115 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:42.727116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:42.727128 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:42.727172 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:42.728061 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.728589 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.729894 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.730204 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.730280 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:42.732590 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:42.732592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:42.732593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:42.732601 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:42.733982 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:42.734438 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:42.734439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:42.734439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:42.734500 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:42.734751 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:42.734751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:42.734752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:42.734850 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:42.734896 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:42.735119 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 09:16:42.735190 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:16:42.735419 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:42.725777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42111 (local address 127.0.98.1:35083)
0620 09:16:42.725986 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:42.726023 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:42.727026 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:42.727261 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:42.727266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:42.727653 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:42.727939 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:42.727957 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.728667 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.728668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:42.729795 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:42.729799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.730330 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.730332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:42.730453 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:42.730962 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:42.730970 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:42.732478 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:42.734083 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:42.734090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:42.734092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:42.734369 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:42.734566 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:42.734567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:42.734568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:42.734700 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:42.734896 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:42.734899 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:42.735024 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 09:16:42.735101 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:16:42.735226 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
I20260620 09:16:42.735927 3004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42111
I20260620 09:16:42.736054 3004 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:42.736202 3004 heartbeater.cc:511] Master 127.0.98.62:42111 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:16:42.736755 2813 ts_manager.cc:194] Registered new tserver with Master: e141deaec6b14b0382f4f2f6d59d6eb0 (127.0.98.1:38389)
I20260620 09:16:42.737389 2813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:16:42.749538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42111
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:42.826193 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:42.826367 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:42.826387 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:42.827458 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:42.827495 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:42.827530 3009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:42.828975 3009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:42.829169 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:42.829360 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:42.833783 3017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:42.833789 3014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:42.833798 3015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:42.833875 3009 server_base.cc:1034] running on GCE node
I20260620 09:16:42.834156 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:42.834354 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:42.835485 3009 hybrid_clock.cc:648] HybridClock initialized: now 1781947002835469 us; error 28 us; skew 500 ppm
Jun 20 09:16:42 dist-test-slave-rq8c krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:42.837757 3009 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:16:42.838158 3009 webserver.cc:460] Webserver started at http://127.0.98.2:38743/ using document root <none> and password file <none>
I20260620 09:16:42.838349 3009 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:42.838395 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:42.838510 3009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:42.839452 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "9b7e45fe328e4976b8ee0f79cebb3508"
format_stamp: "Formatted at 2026-06-20 09:16:42 on dist-test-slave-rq8c"
server_key: "71e99b28d538c28af36bce4f78e1798f"
server_key_iv: "f4ca9a1c1c9ccb0733c79f5c8e222f0f"
server_key_version: "encryptionkey@0"
I20260620 09:16:42.839785 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9b7e45fe328e4976b8ee0f79cebb3508"
format_stamp: "Formatted at 2026-06-20 09:16:42 on dist-test-slave-rq8c"
server_key: "71e99b28d538c28af36bce4f78e1798f"
server_key_iv: "f4ca9a1c1c9ccb0733c79f5c8e222f0f"
server_key_version: "encryptionkey@0"
I20260620 09:16:42.840890 3009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:42.841476 3024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:42.841612 3009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:42.841686 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "9b7e45fe328e4976b8ee0f79cebb3508"
format_stamp: "Formatted at 2026-06-20 09:16:42 on dist-test-slave-rq8c"
server_key: "71e99b28d538c28af36bce4f78e1798f"
server_key_iv: "f4ca9a1c1c9ccb0733c79f5c8e222f0f"
server_key_version: "encryptionkey@0"
I20260620 09:16:42.841743 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:42.854305 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:42.855830 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:42.855944 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:42.856158 3009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:42.856423 3009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:42.856454 3009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:42.856480 3009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:42.856503 3009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:42.861747 3009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33593
I20260620 09:16:42.861778 3137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33593 every 8 connection(s)
I20260620 09:16:42.862049 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:16:42.863380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 3009
I20260620 09:16:42.863456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:16:42.863648 392 external_mini_cluster.cc:1428] Setting key 5bc3b102ff12e8a0d941e46552cb53a5
Jun 20 09:16:42 dist-test-slave-rq8c krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:42.870962 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:42.862649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47897 (local address 127.0.98.62:42111)
0620 09:16:42.862693 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:42.862695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:42.862886 (+ 191us) server_negotiation.cc:407] Connection header received
0620 09:16:42.863867 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:42.863868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:42.863881 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:42.863930 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:42.864502 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.864980 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.865930 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.866188 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.866254 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:42.868474 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:42.868475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:42.868475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:42.868482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:42.869691 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:42.870107 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:42.870107 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:42.870108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:42.870150 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:42.870384 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:42.870384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:42.870385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:42.870471 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:42.870533 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:42.870831 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 09:16:42.870901 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:16:42.871183 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:42.862618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42111 (local address 127.0.98.2:47897)
0620 09:16:42.862779 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:42.862809 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:42.863768 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:42.863999 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:42.864003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:42.864243 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:42.864414 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:42.864418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.865055 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.865056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:42.865844 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:42.865846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.866267 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.866268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:42.866373 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:42.866808 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:42.866815 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:42.868389 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:42.869765 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:42.869772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:42.869774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:42.870040 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:42.870202 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:42.870203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:42.870204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:42.870329 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:42.870525 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:42.870530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:42.870694 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 09:16:42.870807 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 09:16:42.870952 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260620 09:16:42.871810 3138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42111
I20260620 09:16:42.871887 3138 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:42.872052 3138 heartbeater.cc:511] Master 127.0.98.62:42111 requested a full tablet report, sending...
I20260620 09:16:42.872392 2813 ts_manager.cc:194] Registered new tserver with Master: 9b7e45fe328e4976b8ee0f79cebb3508 (127.0.98.2:33593)
I20260620 09:16:42.872728 2813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:16:42.884100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42111
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:42.961730 3143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:42.961913 3143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:42.961934 3143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:42.963030 3143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:42.963069 3143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:42.963097 3143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:42.964574 3143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:42.964774 3143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:42.964967 3143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:42.969513 3151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:42.969483 3149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:42.969476 3148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:42.969571 3143 server_base.cc:1034] running on GCE node
I20260620 09:16:42.969849 3143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:42.970068 3143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:42.971194 3143 hybrid_clock.cc:648] HybridClock initialized: now 1781947002971182 us; error 34 us; skew 500 ppm
Jun 20 09:16:42 dist-test-slave-rq8c krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:42.973487 3143 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:16:42.973867 3143 webserver.cc:460] Webserver started at http://127.0.98.3:32777/ using document root <none> and password file <none>
I20260620 09:16:42.974048 3143 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:42.974088 3143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:42.974206 3143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:42.975145 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "6fa6fc008bf4467194197d427ca05f46"
format_stamp: "Formatted at 2026-06-20 09:16:42 on dist-test-slave-rq8c"
server_key: "1d73803439dacd19c282def83c2751bd"
server_key_iv: "19e3baa2971d6e32605ff4f4ef290740"
server_key_version: "encryptionkey@0"
I20260620 09:16:42.975466 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6fa6fc008bf4467194197d427ca05f46"
format_stamp: "Formatted at 2026-06-20 09:16:42 on dist-test-slave-rq8c"
server_key: "1d73803439dacd19c282def83c2751bd"
server_key_iv: "19e3baa2971d6e32605ff4f4ef290740"
server_key_version: "encryptionkey@0"
I20260620 09:16:42.976671 3143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:42.977311 3158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:42.977450 3143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:42.977525 3143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "6fa6fc008bf4467194197d427ca05f46"
format_stamp: "Formatted at 2026-06-20 09:16:42 on dist-test-slave-rq8c"
server_key: "1d73803439dacd19c282def83c2751bd"
server_key_iv: "19e3baa2971d6e32605ff4f4ef290740"
server_key_version: "encryptionkey@0"
I20260620 09:16:42.977592 3143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:42.985481 3143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:42.986999 3143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:42.987111 3143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:42.987295 3143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:42.987564 3143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:42.987596 3143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:42.987637 3143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:42.987656 3143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:42.992805 3143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34963
I20260620 09:16:42.992839 3271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34963 every 8 connection(s)
I20260620 09:16:42.993103 3143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:16:42.997671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 3143
I20260620 09:16:42.997749 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:16:42.997905 392 external_mini_cluster.cc:1428] Setting key 3759aa1e13f0e733e8a8f4d2160d7b97
Jun 20 09:16:42 dist-test-slave-rq8c krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:43.001384 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:42.993711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54273 (local address 127.0.98.62:42111)
0620 09:16:42.993752 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:42.993754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:42.993910 (+ 156us) server_negotiation.cc:407] Connection header received
0620 09:16:42.994793 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:42.994794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:42.994803 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:42.994843 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:42.995385 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.995903 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.996824 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.996996 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.997038 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:42.999091 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:42.999093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:42.999093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:42.999100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:43.000217 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:43.000554 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:43.000555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:43.000555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:43.000606 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:43.000815 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:43.000816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:43.000817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:43.000925 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:43.000978 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:43.001222 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 09:16:43.001294 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:16:43.001547 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:42.993668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42111 (local address 127.0.98.3:54273)
0620 09:16:42.993808 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:42.993846 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:42.994696 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:42.994911 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:42.994914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:42.995150 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:42.995324 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:42.995329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.995971 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.995972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:42.996766 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:42.996768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:42.997046 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:42.997046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:42.997133 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:42.997584 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:42.997591 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:42.999009 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:43.000284 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:43.000290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:43.000292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:43.000491 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:43.000664 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:43.000664 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:43.000665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:43.000764 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:43.000976 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:43.000981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:43.001108 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 09:16:43.001194 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:16:43.001342 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260620 09:16:43.001947 3272 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42111
I20260620 09:16:43.002032 3272 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:43.002168 3272 heartbeater.cc:511] Master 127.0.98.62:42111 requested a full tablet report, sending...
I20260620 09:16:43.002496 2813 ts_manager.cc:194] Registered new tserver with Master: 6fa6fc008bf4467194197d427ca05f46 (127.0.98.3:34963)
I20260620 09:16:43.002857 2813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54273
I20260620 09:16:43.009354 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 09:16:43.018733 2853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:16:43.017939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.0.98.62:42111)
0620 09:16:43.017990 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:43.017992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:43.018073 (+ 81us) server_negotiation.cc:407] Connection header received
0620 09:16:43.018134 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:43.018135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:43.018146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:43.018205 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:43.018672 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34062: BlockingRecv error: recv got EOF from 127.0.0.1:34062 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:16:43.019595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 2875
I20260620 09:16:43.023518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 3009
I20260620 09:16:43.026950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 3143
I20260620 09:16:43.030400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 2781
2026-06-20T09:16:43Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1642 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:16:43 dist-test-slave-rq8c krb5kdc[3284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:16:43 dist-test-slave-rq8c krb5kdc[3284](info): set up 2 sockets
Jun 20 09:16:43 dist-test-slave-rq8c krb5kdc[3284](info): commencing operation
krb5kdc: starting...
W20260620 09:16:43.979202 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:16:44 dist-test-slave-rq8c krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:16:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:16:44.086666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41971
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:44.164003 3297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:44.164209 3297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:44.164238 3297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:44.165354 3297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:44.165391 3297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:44.165409 3297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:44.165426 3297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:44.165441 3297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:44.166880 3297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:44.167111 3297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:44.167304 3297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:44.171849 3305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:44.171867 3303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:44.171872 3297 server_base.cc:1034] running on GCE node
W20260620 09:16:44.171851 3302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:44.172257 3297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:44.172513 3297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:44.173660 3297 hybrid_clock.cc:648] HybridClock initialized: now 1781947004173648 us; error 31 us; skew 500 ppm
Jun 20 09:16:44 dist-test-slave-rq8c krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:44.179102 3297 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:16:44.179512 3297 webserver.cc:460] Webserver started at http://127.0.98.62:40185/ using document root <none> and password file <none>
I20260620 09:16:44.179749 3297 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:44.179800 3297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:44.179917 3297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:44.180971 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "c8fb1aa41ca44f65b1b6e12f8739b807"
format_stamp: "Formatted at 2026-06-20 09:16:44 on dist-test-slave-rq8c"
server_key: "0fa83481a55499bec2957127f2fa2da4"
server_key_iv: "f2e86e25c40dba85833ebf7ed0031d97"
server_key_version: "encryptionkey@0"
I20260620 09:16:44.181296 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "c8fb1aa41ca44f65b1b6e12f8739b807"
format_stamp: "Formatted at 2026-06-20 09:16:44 on dist-test-slave-rq8c"
server_key: "0fa83481a55499bec2957127f2fa2da4"
server_key_iv: "f2e86e25c40dba85833ebf7ed0031d97"
server_key_version: "encryptionkey@0"
I20260620 09:16:44.182488 3297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:44.183118 3312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:44.183269 3297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:44.183348 3297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "c8fb1aa41ca44f65b1b6e12f8739b807"
format_stamp: "Formatted at 2026-06-20 09:16:44 on dist-test-slave-rq8c"
server_key: "0fa83481a55499bec2957127f2fa2da4"
server_key_iv: "f2e86e25c40dba85833ebf7ed0031d97"
server_key_version: "encryptionkey@0"
I20260620 09:16:44.183408 3297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:44.191011 3297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:44.192673 3297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:44.192796 3297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:44.196316 3297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41971
I20260620 09:16:44.196352 3364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41971 every 8 connection(s)
I20260620 09:16:44.196641 3297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:44.197372 3365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:44.199478 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807: Bootstrap starting.
I20260620 09:16:44.200022 3365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:44.200280 3365 log.cc:824] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:44.200867 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807: No bootstrap required, opened a new log
I20260620 09:16:44.201335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 3297
I20260620 09:16:44.201439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:44.201615 392 external_mini_cluster.cc:1428] Setting key 25821eab8f7eb394e8bf5b0dd8d0078e
I20260620 09:16:44.203130 3365 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fb1aa41ca44f65b1b6e12f8739b807" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41971 } }
I20260620 09:16:44.203262 3365 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:44.203293 3365 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8fb1aa41ca44f65b1b6e12f8739b807, State: Initialized, Role: FOLLOWER
I20260620 09:16:44.203426 3365 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fb1aa41ca44f65b1b6e12f8739b807" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41971 } }
I20260620 09:16:44.203487 3365 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:44.203516 3365 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:44.203562 3365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:44.204046 3365 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fb1aa41ca44f65b1b6e12f8739b807" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41971 } }
I20260620 09:16:44.204137 3365 leader_election.cc:302] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8fb1aa41ca44f65b1b6e12f8739b807; no voters:
I20260620 09:16:44.204276 3365 leader_election.cc:288] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:44.204314 3370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:44.204437 3370 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [term 1 LEADER]: Becoming Leader. State: Replica: c8fb1aa41ca44f65b1b6e12f8739b807, State: Running, Role: LEADER
I20260620 09:16:44.204516 3365 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:44.204588 3370 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fb1aa41ca44f65b1b6e12f8739b807" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41971 } }
I20260620 09:16:44.204895 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8fb1aa41ca44f65b1b6e12f8739b807. Latest consensus state: current_term: 1 leader_uuid: "c8fb1aa41ca44f65b1b6e12f8739b807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fb1aa41ca44f65b1b6e12f8739b807" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41971 } } }
I20260620 09:16:44.204967 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:44.205096 3372 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8fb1aa41ca44f65b1b6e12f8739b807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8fb1aa41ca44f65b1b6e12f8739b807" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41971 } } }
I20260620 09:16:44.205154 3372 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:44.205265 3377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:44.205932 3377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:44.207311 3377 catalog_manager.cc:1347] Generated new cluster ID: 3b98a9c7441e42ecbbc180d3f635476d
I20260620 09:16:44.207365 3377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:16:44 dist-test-slave-rq8c krb5kdc[3284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:44.211810 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:44.202048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.0.98.62:41971)
0620 09:16:44.202225 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:44.202234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:44.202259 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:16:44.203070 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:44.203077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:44.203123 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:44.203243 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:44.203906 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:44.205123 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:44.206194 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:44.206766 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:44.206867 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:44.209069 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:44.209072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:44.209073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:44.209082 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:44.210515 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:44.210818 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:44.210821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:44.210822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:44.210896 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:44.211060 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:44.211060 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:44.211061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:44.211276 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:44.211410 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:44.211515 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 09:16:44.211669 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260620 09:16:44.212566 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:16:44.228430 3377 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:44.228938 3377 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:44.230595 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:44.226761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.0.98.62:41971)
0620 09:16:44.226808 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:44.226810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:44.226926 (+ 116us) server_negotiation.cc:407] Connection header received
0620 09:16:44.226990 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:44.226991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:44.227001 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:44.227047 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:44.227439 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:44.227987 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:44.229079 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:44.229389 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:44.229458 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:44.230138 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:44.230139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:44.230139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:44.230347 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:44.230366 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:44.230372 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:44.230422 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:44.230481 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 09:16:44.230521 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:16:44.232967 3377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8fb1aa41ca44f65b1b6e12f8739b807: Generated new TSK 0
I20260620 09:16:44.233137 3377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:16:44.236452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 3297
2026-06-20T09:16:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1205 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:16:44 dist-test-slave-rq8c krb5kdc[3398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:16:44 dist-test-slave-rq8c krb5kdc[3398](info): set up 2 sockets
Jun 20 09:16:44 dist-test-slave-rq8c krb5kdc[3398](info): commencing operation
krb5kdc: starting...
W20260620 09:16:45.191231 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:16:45 dist-test-slave-rq8c krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:16:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:16:45.299883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36615
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:45.377684 3411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:45.377873 3411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:45.377902 3411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:45.379027 3411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:45.379065 3411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:45.379083 3411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:45.379099 3411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:45.379115 3411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:45.380548 3411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:45.380774 3411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:45.380957 3411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:45.385372 3419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:45.385399 3416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:45.385461 3411 server_base.cc:1034] running on GCE node
W20260620 09:16:45.385428 3417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:45.385828 3411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:45.386080 3411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:45.387212 3411 hybrid_clock.cc:648] HybridClock initialized: now 1781947005387201 us; error 28 us; skew 500 ppm
Jun 20 09:16:45 dist-test-slave-rq8c krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:45.389441 3411 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:16:45.389819 3411 webserver.cc:460] Webserver started at http://127.0.98.62:36957/ using document root <none> and password file <none>
I20260620 09:16:45.389987 3411 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:45.390025 3411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:45.390110 3411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:45.391037 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "d520bdd9f7c54880b5391b7d66ccf6eb"
format_stamp: "Formatted at 2026-06-20 09:16:45 on dist-test-slave-rq8c"
server_key: "7792e35960fec454ac5d8290f7a14ece"
server_key_iv: "67209bbe42c7f777b62e733c0dcf82a3"
server_key_version: "encryptionkey@0"
I20260620 09:16:45.391307 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "d520bdd9f7c54880b5391b7d66ccf6eb"
format_stamp: "Formatted at 2026-06-20 09:16:45 on dist-test-slave-rq8c"
server_key: "7792e35960fec454ac5d8290f7a14ece"
server_key_iv: "67209bbe42c7f777b62e733c0dcf82a3"
server_key_version: "encryptionkey@0"
I20260620 09:16:45.392483 3411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:16:45.393160 3426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:45.393334 3411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:45.393412 3411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "d520bdd9f7c54880b5391b7d66ccf6eb"
format_stamp: "Formatted at 2026-06-20 09:16:45 on dist-test-slave-rq8c"
server_key: "7792e35960fec454ac5d8290f7a14ece"
server_key_iv: "67209bbe42c7f777b62e733c0dcf82a3"
server_key_version: "encryptionkey@0"
I20260620 09:16:45.393469 3411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:45.406595 3411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:45.408627 3411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:45.408752 3411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:45.412364 3411 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36615
I20260620 09:16:45.412420 3478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36615 every 8 connection(s)
I20260620 09:16:45.412711 3411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:45.413623 3479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:45.413872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 3411
I20260620 09:16:45.413954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:45.414116 392 external_mini_cluster.cc:1428] Setting key 5db8c9734ad4ee7e8677a8badd8b64e4
I20260620 09:16:45.416280 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb: Bootstrap starting.
I20260620 09:16:45.416827 3479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:45.417106 3479 log.cc:824] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:45.417773 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb: No bootstrap required, opened a new log
I20260620 09:16:45.419979 3479 raft_consensus.cc:348] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d520bdd9f7c54880b5391b7d66ccf6eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36615 } }
I20260620 09:16:45.420107 3479 raft_consensus.cc:374] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:45.420128 3479 raft_consensus.cc:729] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d520bdd9f7c54880b5391b7d66ccf6eb, State: Initialized, Role: FOLLOWER
Jun 20 09:16:45 dist-test-slave-rq8c krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:45.420233 3479 consensus_queue.cc:260] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d520bdd9f7c54880b5391b7d66ccf6eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36615 } }
I20260620 09:16:45.420313 3479 raft_consensus.cc:388] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:45.420359 3479 raft_consensus.cc:482] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:45.420399 3479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:45.420964 3479 raft_consensus.cc:504] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d520bdd9f7c54880b5391b7d66ccf6eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36615 } }
I20260620 09:16:45.421063 3479 leader_election.cc:302] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d520bdd9f7c54880b5391b7d66ccf6eb; no voters:
I20260620 09:16:45.421204 3479 leader_election.cc:288] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:45.421280 3484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:45.421444 3484 raft_consensus.cc:686] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [term 1 LEADER]: Becoming Leader. State: Replica: d520bdd9f7c54880b5391b7d66ccf6eb, State: Running, Role: LEADER
I20260620 09:16:45.421468 3479 sys_catalog.cc:564] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:45.421797 3484 consensus_queue.cc:237] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d520bdd9f7c54880b5391b7d66ccf6eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36615 } }
I20260620 09:16:45.422199 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d520bdd9f7c54880b5391b7d66ccf6eb. Latest consensus state: current_term: 1 leader_uuid: "d520bdd9f7c54880b5391b7d66ccf6eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d520bdd9f7c54880b5391b7d66ccf6eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36615 } } }
I20260620 09:16:45.422273 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:45.422581 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d520bdd9f7c54880b5391b7d66ccf6eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d520bdd9f7c54880b5391b7d66ccf6eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36615 } } }
I20260620 09:16:45.422638 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:45.422910 3492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:45.423446 3492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:45.424158 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:45.414578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.0.98.62:36615)
0620 09:16:45.414724 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:45.414732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:45.414755 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:16:45.415158 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:45.415164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:45.415208 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:45.415315 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:45.416040 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.416892 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.417802 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.418220 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.418297 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:45.420717 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:45.420721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:45.420721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:45.420732 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:45.422566 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:45.422983 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:45.422986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:45.422987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:45.423080 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:45.423283 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:45.423284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:45.423284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:45.423566 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:45.423738 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:45.423837 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 09:16:45.423967 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260620 09:16:45.424989 3492 catalog_manager.cc:1347] Generated new cluster ID: 4136ffbde0944b2fb223f15ea1ff4f0f
I20260620 09:16:45.425036 3492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:16:45.444448 3492 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:45.444963 3492 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:45.448836 3492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d520bdd9f7c54880b5391b7d66ccf6eb: Generated new TSK 0
I20260620 09:16:45.449013 3492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:16:45.470571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36615
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:45.548467 3505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:45.548662 3505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:45.548683 3505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:45.549763 3505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:45.549796 3505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:45.549820 3505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:45.551165 3505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:45.551349 3505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:45.551525 3505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:16:45.556262 3505 server_base.cc:1034] running on GCE node
W20260620 09:16:45.556217 3510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:45.556219 3513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:45.556253 3511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:45.556555 3505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:45.556772 3505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:45.557916 3505 hybrid_clock.cc:648] HybridClock initialized: now 1781947005557896 us; error 36 us; skew 500 ppm
Jun 20 09:16:45 dist-test-slave-rq8c krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:45.560256 3505 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:16:45.560638 3505 webserver.cc:460] Webserver started at http://127.0.98.1:38155/ using document root <none> and password file <none>
I20260620 09:16:45.560828 3505 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:45.560884 3505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:45.561007 3505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:45.561977 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "ebafcef3b4cf4a109260791584dae074"
format_stamp: "Formatted at 2026-06-20 09:16:45 on dist-test-slave-rq8c"
server_key: "d897a5941045dbf8ad5d82af86e30392"
server_key_iv: "3c87c0b4b89f4c4d1a9206e5e12e1acc"
server_key_version: "encryptionkey@0"
I20260620 09:16:45.562288 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ebafcef3b4cf4a109260791584dae074"
format_stamp: "Formatted at 2026-06-20 09:16:45 on dist-test-slave-rq8c"
server_key: "d897a5941045dbf8ad5d82af86e30392"
server_key_iv: "3c87c0b4b89f4c4d1a9206e5e12e1acc"
server_key_version: "encryptionkey@0"
I20260620 09:16:45.563448 3505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:45.564121 3520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:45.564265 3505 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:45.564344 3505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "ebafcef3b4cf4a109260791584dae074"
format_stamp: "Formatted at 2026-06-20 09:16:45 on dist-test-slave-rq8c"
server_key: "d897a5941045dbf8ad5d82af86e30392"
server_key_iv: "3c87c0b4b89f4c4d1a9206e5e12e1acc"
server_key_version: "encryptionkey@0"
I20260620 09:16:45.564409 3505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:45.574558 3505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:45.576294 3505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:45.576428 3505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:45.576627 3505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:45.576897 3505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:45.576929 3505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:45.576956 3505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:45.576978 3505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:45.582289 3505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35641
I20260620 09:16:45.582345 3633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35641 every 8 connection(s)
I20260620 09:16:45.582602 3505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:16:45.584721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 3505
I20260620 09:16:45.584826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:16:45.585031 392 external_mini_cluster.cc:1428] Setting key f2bd8fbe3a6ff1d28777a885acc929b8
Jun 20 09:16:45 dist-test-slave-rq8c krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:45.592363 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:45.583190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50497 (local address 127.0.98.62:36615)
0620 09:16:45.583239 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:45.583240 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:45.583440 (+ 200us) server_negotiation.cc:407] Connection header received
0620 09:16:45.584087 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:45.584088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:45.584098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:45.584145 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:45.585296 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.585984 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.587340 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.587680 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.587743 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:45.589861 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:45.589862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:45.589863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:45.589869 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:45.591072 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:45.591489 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:45.591490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:45.591490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:45.591533 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:45.591803 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:45.591803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:45.591803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:45.591897 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:45.591941 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:45.592243 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 09:16:45.592300 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:16:45.592600 3636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:45.583185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36615 (local address 127.0.98.1:50497)
0620 09:16:45.583312 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:45.583360 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:45.583979 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:45.584210 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:45.584213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:45.584914 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:45.585195 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:45.585202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.586082 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.586084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:45.587257 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:45.587260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.587757 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.587758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:45.587863 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:45.588317 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:45.588325 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:45.589776 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:45.591150 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:45.591156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:45.591158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:45.591422 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:45.591584 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:45.591585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:45.591585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:45.591733 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:45.591954 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:45.591959 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:45.592120 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 09:16:45.592235 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 09:16:45.592384 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260620 09:16:45.593046 3634 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36615
I20260620 09:16:45.593128 3634 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:45.593302 3634 heartbeater.cc:511] Master 127.0.98.62:36615 requested a full tablet report, sending...
I20260620 09:16:45.593763 3443 ts_manager.cc:194] Registered new tserver with Master: ebafcef3b4cf4a109260791584dae074 (127.0.98.1:35641)
I20260620 09:16:45.594336 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:16:45.607339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36615
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:45.683197 3639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:45.683393 3639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:45.683414 3639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:45.684513 3639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:45.684545 3639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:45.684569 3639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:45.685899 3639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:45.686067 3639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:45.686237 3639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:45.690769 3645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:45.690779 3644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:45.690862 3639 server_base.cc:1034] running on GCE node
W20260620 09:16:45.690995 3647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:45.691177 3639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:45.691423 3639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:45.692561 3639 hybrid_clock.cc:648] HybridClock initialized: now 1781947005692550 us; error 28 us; skew 500 ppm
Jun 20 09:16:45 dist-test-slave-rq8c krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:45.694772 3639 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:16:45.695173 3639 webserver.cc:460] Webserver started at http://127.0.98.2:41601/ using document root <none> and password file <none>
I20260620 09:16:45.695369 3639 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:45.695415 3639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:45.695530 3639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:45.696601 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "ee8d820bc9fd4f7dbe76ebbdbdf41cc4"
format_stamp: "Formatted at 2026-06-20 09:16:45 on dist-test-slave-rq8c"
server_key: "3e1d976a49f481b427929aaa14d550b0"
server_key_iv: "7067fea2d4530c970fc0f902f75cd64c"
server_key_version: "encryptionkey@0"
I20260620 09:16:45.696913 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ee8d820bc9fd4f7dbe76ebbdbdf41cc4"
format_stamp: "Formatted at 2026-06-20 09:16:45 on dist-test-slave-rq8c"
server_key: "3e1d976a49f481b427929aaa14d550b0"
server_key_iv: "7067fea2d4530c970fc0f902f75cd64c"
server_key_version: "encryptionkey@0"
I20260620 09:16:45.698115 3639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:16:45.698707 3654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:45.698853 3639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:45.698927 3639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "ee8d820bc9fd4f7dbe76ebbdbdf41cc4"
format_stamp: "Formatted at 2026-06-20 09:16:45 on dist-test-slave-rq8c"
server_key: "3e1d976a49f481b427929aaa14d550b0"
server_key_iv: "7067fea2d4530c970fc0f902f75cd64c"
server_key_version: "encryptionkey@0"
I20260620 09:16:45.698980 3639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:45.749603 3639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:45.751765 3639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:45.751873 3639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:45.752053 3639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:45.752346 3639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:45.752380 3639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:45.752403 3639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:45.752426 3639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:45.757696 3639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41913
I20260620 09:16:45.757759 3767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41913 every 8 connection(s)
I20260620 09:16:45.758054 3639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:16:45.761889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 3639
I20260620 09:16:45.762005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:16:45.762200 392 external_mini_cluster.cc:1428] Setting key 1437bd4063deab9e0db8b0803eff7a9a
Jun 20 09:16:45 dist-test-slave-rq8c krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:45.767031 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:45.758726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38323 (local address 127.0.98.62:36615)
0620 09:16:45.758802 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:45.758804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:45.758974 (+ 170us) server_negotiation.cc:407] Connection header received
0620 09:16:45.759431 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:45.759432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:45.759440 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:45.759490 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:45.760404 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.760894 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.762087 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.762318 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.762388 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:45.764405 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:45.764406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:45.764407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:45.764414 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:45.765698 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:45.766097 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:45.766099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:45.766099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:45.766161 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:45.766391 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:45.766391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:45.766392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:45.766506 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:45.766569 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:45.766852 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 09:16:45.766937 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:16:45.767197 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:45.758693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36615 (local address 127.0.98.2:38323)
0620 09:16:45.758857 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:45.758894 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:45.759356 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:45.759546 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:45.759549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:45.760143 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:45.760333 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:45.760339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.760965 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.760966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:45.761952 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:45.761956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.762401 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.762402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:45.762541 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:45.763027 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:45.763034 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:45.764326 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:45.765793 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:45.765800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:45.765802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:45.766025 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:45.766227 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:45.766228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:45.766228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:45.766329 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:45.766565 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:45.766570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:45.766701 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 09:16:45.766829 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 09:16:45.766986 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260620 09:16:45.767818 3768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36615
I20260620 09:16:45.767915 3768 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:45.768085 3768 heartbeater.cc:511] Master 127.0.98.62:36615 requested a full tablet report, sending...
I20260620 09:16:45.768472 3443 ts_manager.cc:194] Registered new tserver with Master: ee8d820bc9fd4f7dbe76ebbdbdf41cc4 (127.0.98.2:41913)
I20260620 09:16:45.768949 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38323
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:16:45.783951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36615
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:45.861418 3773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:45.861615 3773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:45.861636 3773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:45.862756 3773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:45.862795 3773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:45.862819 3773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:45.864195 3773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:45.864396 3773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:45.864586 3773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:45.869195 3779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:45.869314 3778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:45.869211 3773 server_base.cc:1034] running on GCE node
W20260620 09:16:45.869211 3781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:45.869628 3773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:45.869853 3773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:45.870992 3773 hybrid_clock.cc:648] HybridClock initialized: now 1781947005870978 us; error 26 us; skew 500 ppm
Jun 20 09:16:45 dist-test-slave-rq8c krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:45.873281 3773 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:16:45.873680 3773 webserver.cc:460] Webserver started at http://127.0.98.3:37505/ using document root <none> and password file <none>
I20260620 09:16:45.873883 3773 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:45.873929 3773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:45.874059 3773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:45.875036 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "f296a07c5ab44ba8b7aaf456dd994fea"
format_stamp: "Formatted at 2026-06-20 09:16:45 on dist-test-slave-rq8c"
server_key: "750204f3d7fa8c69fa6e9aafbfa2e742"
server_key_iv: "1257ef23d740fe78dabf012e353d2602"
server_key_version: "encryptionkey@0"
I20260620 09:16:45.875346 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f296a07c5ab44ba8b7aaf456dd994fea"
format_stamp: "Formatted at 2026-06-20 09:16:45 on dist-test-slave-rq8c"
server_key: "750204f3d7fa8c69fa6e9aafbfa2e742"
server_key_iv: "1257ef23d740fe78dabf012e353d2602"
server_key_version: "encryptionkey@0"
I20260620 09:16:45.876592 3773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:16:45.877250 3788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:45.877395 3773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:45.877475 3773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "f296a07c5ab44ba8b7aaf456dd994fea"
format_stamp: "Formatted at 2026-06-20 09:16:45 on dist-test-slave-rq8c"
server_key: "750204f3d7fa8c69fa6e9aafbfa2e742"
server_key_iv: "1257ef23d740fe78dabf012e353d2602"
server_key_version: "encryptionkey@0"
I20260620 09:16:45.877540 3773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:45.905058 3773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:45.906827 3773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:45.906955 3773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:45.907167 3773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:45.907461 3773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:45.907495 3773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:45.907522 3773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:45.907546 3773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:45.912837 3773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35771
I20260620 09:16:45.912884 3901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35771 every 8 connection(s)
I20260620 09:16:45.913169 3773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:16:45.918464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 3773
I20260620 09:16:45.918558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:16:45.918736 392 external_mini_cluster.cc:1428] Setting key 5f282ed9fdd0a643d044b0859588cd68
Jun 20 09:16:45 dist-test-slave-rq8c krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:45.922032 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:45.913836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51325 (local address 127.0.98.62:36615)
0620 09:16:45.913894 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:45.913895 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:45.914102 (+ 207us) server_negotiation.cc:407] Connection header received
0620 09:16:45.914594 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:45.914595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:45.914606 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:45.914658 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:45.915579 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.916108 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.917065 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.917269 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.917332 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:45.919449 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:45.919451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:45.919451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:45.919458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:45.920721 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:45.921093 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:45.921094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:45.921095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:45.921153 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:45.921367 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:45.921367 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:45.921368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:45.921480 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:45.921534 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:45.921894 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 09:16:45.921969 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 09:16:45.922463 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:45.913823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36615 (local address 127.0.98.3:51325)
0620 09:16:45.913982 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:45.914021 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:45.914504 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:45.914727 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:45.914730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:45.915292 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:45.915487 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:45.915492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.916188 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.916189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:45.916988 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:45.916990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.917342 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.917343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:45.917456 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:45.917884 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:45.917891 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:45.919365 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:45.920793 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:45.920799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:45.920801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:45.921027 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:45.921205 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:45.921205 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:45.921206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:45.921313 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:45.921533 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:45.921539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:45.921702 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 09:16:45.922124 (+ 422us) client_negotiation.cc:240] Negotiation successful
0620 09:16:45.922296 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260620 09:16:45.922953 3902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36615
I20260620 09:16:45.923053 3902 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:45.923225 3902 heartbeater.cc:511] Master 127.0.98.62:36615 requested a full tablet report, sending...
I20260620 09:16:45.923585 3443 ts_manager.cc:194] Registered new tserver with Master: f296a07c5ab44ba8b7aaf456dd994fea (127.0.98.3:35771)
I20260620 09:16:45.923965 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51325
I20260620 09:16:45.930459 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:16:45 dist-test-slave-rq8c krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947005, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 09:16:45 dist-test-slave-rq8c krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947005, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:45.951253 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:45.944557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.0.98.62:36615)
0620 09:16:45.944616 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:45.944619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:45.944726 (+ 107us) server_negotiation.cc:407] Connection header received
0620 09:16:45.944778 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:45.944779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:45.944788 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:45.944839 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:45.945334 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.945815 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.946564 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:45.946740 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:45.946788 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:45.948994 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:45.948995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:45.948995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:45.949002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:45.950332 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:45.950644 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:45.950645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:45.950645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:45.950692 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:45.950818 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:45.950819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:45.950819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:45.950958 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:45.951029 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:45.951131 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 09:16:45.951187 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 09:16:45.951529 3443 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39558
I20260620 09:16:45.951915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 3505
I20260620 09:16:45.955506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 3639
I20260620 09:16:45.959030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 3773
I20260620 09:16:45.962483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 3411
2026-06-20T09:16:45Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1726 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:16:46 dist-test-slave-rq8c krb5kdc[3914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:16:46 dist-test-slave-rq8c krb5kdc[3914](info): set up 2 sockets
Jun 20 09:16:46 dist-test-slave-rq8c krb5kdc[3914](info): commencing operation
krb5kdc: starting...
W20260620 09:16:46.915274 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:16:46 dist-test-slave-rq8c krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:16:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:16:47.020066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37219
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:47.097610 3927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:47.097895 3927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:47.097929 3927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:47.099591 3927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:47.099658 3927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:47.099678 3927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:47.099704 3927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:47.099726 3927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:47.101226 3927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:47.101429 3927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:47.101598 3927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:47.106069 3932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:47.106146 3933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:47.106070 3935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:47.106235 3927 server_base.cc:1034] running on GCE node
I20260620 09:16:47.106477 3927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:47.106758 3927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:47.107888 3927 hybrid_clock.cc:648] HybridClock initialized: now 1781947007107878 us; error 35 us; skew 500 ppm
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:47.110319 3927 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:16:47.110714 3927 webserver.cc:460] Webserver started at http://127.0.98.62:43437/ using document root <none> and password file <none>
I20260620 09:16:47.110898 3927 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:47.110936 3927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:47.111023 3927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:47.112061 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "a574e48ad49e491abe106835e89345b9"
format_stamp: "Formatted at 2026-06-20 09:16:47 on dist-test-slave-rq8c"
server_key: "8a7577c1ad127b61431ff761267124a1"
server_key_iv: "85da4013e54d52029f758d20642a4531"
server_key_version: "encryptionkey@0"
I20260620 09:16:47.112349 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "a574e48ad49e491abe106835e89345b9"
format_stamp: "Formatted at 2026-06-20 09:16:47 on dist-test-slave-rq8c"
server_key: "8a7577c1ad127b61431ff761267124a1"
server_key_iv: "85da4013e54d52029f758d20642a4531"
server_key_version: "encryptionkey@0"
I20260620 09:16:47.113456 3927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:47.114090 3942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:47.114220 3927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:47.114296 3927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "a574e48ad49e491abe106835e89345b9"
format_stamp: "Formatted at 2026-06-20 09:16:47 on dist-test-slave-rq8c"
server_key: "8a7577c1ad127b61431ff761267124a1"
server_key_iv: "85da4013e54d52029f758d20642a4531"
server_key_version: "encryptionkey@0"
I20260620 09:16:47.114359 3927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:47.127790 3927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:47.129787 3927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:47.129906 3927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:47.133440 3927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37219
I20260620 09:16:47.133483 3994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37219 every 8 connection(s)
I20260620 09:16:47.133782 3927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:47.133915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 3927
I20260620 09:16:47.134016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:47.134238 392 external_mini_cluster.cc:1428] Setting key a05f5deb8738514b6935dd4b0c5b0e8b
I20260620 09:16:47.134758 3995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:47.137933 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9: Bootstrap starting.
I20260620 09:16:47.138553 3995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:47.138850 3995 log.cc:824] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:47.139562 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9: No bootstrap required, opened a new log
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:47.141624 3995 raft_consensus.cc:348] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a574e48ad49e491abe106835e89345b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37219 } }
I20260620 09:16:47.141778 3995 raft_consensus.cc:374] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:47.141808 3995 raft_consensus.cc:729] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a574e48ad49e491abe106835e89345b9, State: Initialized, Role: FOLLOWER
I20260620 09:16:47.141934 3995 consensus_queue.cc:260] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a574e48ad49e491abe106835e89345b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37219 } }
I20260620 09:16:47.142016 3995 raft_consensus.cc:388] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:47.142055 3995 raft_consensus.cc:482] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:47.142102 3995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:47.142688 3995 raft_consensus.cc:504] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a574e48ad49e491abe106835e89345b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37219 } }
I20260620 09:16:47.142802 3995 leader_election.cc:302] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a574e48ad49e491abe106835e89345b9; no voters:
I20260620 09:16:47.142961 3995 leader_election.cc:288] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:47.143033 4000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:47.143199 3995 sys_catalog.cc:564] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:47.143229 4000 raft_consensus.cc:686] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [term 1 LEADER]: Becoming Leader. State: Replica: a574e48ad49e491abe106835e89345b9, State: Running, Role: LEADER
I20260620 09:16:47.143311 4000 consensus_queue.cc:237] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a574e48ad49e491abe106835e89345b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37219 } }
I20260620 09:16:47.143610 4001 sys_catalog.cc:455] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a574e48ad49e491abe106835e89345b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a574e48ad49e491abe106835e89345b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37219 } } }
I20260620 09:16:47.143710 4001 sys_catalog.cc:458] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:47.143860 4002 sys_catalog.cc:455] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a574e48ad49e491abe106835e89345b9. Latest consensus state: current_term: 1 leader_uuid: "a574e48ad49e491abe106835e89345b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a574e48ad49e491abe106835e89345b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37219 } } }
I20260620 09:16:47.143932 4002 sys_catalog.cc:458] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:47.144052 4007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:47.144515 4007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:47.145016 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:47.134735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.0.98.62:37219)
0620 09:16:47.134932 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:47.134941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:47.134971 (+ 30us) server_negotiation.cc:407] Connection header received
0620 09:16:47.135484 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:47.135492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:47.135554 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:47.135739 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:47.136267 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.137560 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.138350 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.138945 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.139073 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:47.141396 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:47.141401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:47.141401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:47.141412 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:47.143246 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:47.143669 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:47.143672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:47.143672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:47.143781 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:47.144011 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:47.144012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:47.144012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:47.144345 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:47.144507 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:47.144671 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 09:16:47.144796 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260620 09:16:47.145931 4007 catalog_manager.cc:1347] Generated new cluster ID: 1f31d3b6af514c3fbac1eeb03a71b8c8
I20260620 09:16:47.145979 4007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:16:47.181339 4007 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:47.182073 4007 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:47.193163 4007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a574e48ad49e491abe106835e89345b9: Generated new TSK 0
I20260620 09:16:47.193347 4007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:16:47.220227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37219
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:47.325974 4021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:47.326177 4021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:47.326200 4021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:47.327306 4021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:47.327342 4021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:47.327368 4021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:47.328807 4021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:47.329020 4021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:47.329234 4021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:47.333961 4029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:47.334002 4027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:47.333933 4026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:47.334004 4021 server_base.cc:1034] running on GCE node
I20260620 09:16:47.334304 4021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:47.334528 4021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:47.335671 4021 hybrid_clock.cc:648] HybridClock initialized: now 1781947007335655 us; error 31 us; skew 500 ppm
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:47.338228 4021 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:16:47.338626 4021 webserver.cc:460] Webserver started at http://127.0.98.1:42695/ using document root <none> and password file <none>
I20260620 09:16:47.338836 4021 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:47.338883 4021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:47.339001 4021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:47.340024 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a43bc49820743f18d817999e83a1734"
format_stamp: "Formatted at 2026-06-20 09:16:47 on dist-test-slave-rq8c"
server_key: "8b66baadd136ad8a85c5f4ac4071ca0c"
server_key_iv: "b7e3dadbc5f8dfe796d315e8bab49150"
server_key_version: "encryptionkey@0"
I20260620 09:16:47.340328 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a43bc49820743f18d817999e83a1734"
format_stamp: "Formatted at 2026-06-20 09:16:47 on dist-test-slave-rq8c"
server_key: "8b66baadd136ad8a85c5f4ac4071ca0c"
server_key_iv: "b7e3dadbc5f8dfe796d315e8bab49150"
server_key_version: "encryptionkey@0"
I20260620 09:16:47.341449 4021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:47.342044 4036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:47.342180 4021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:47.342257 4021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "8a43bc49820743f18d817999e83a1734"
format_stamp: "Formatted at 2026-06-20 09:16:47 on dist-test-slave-rq8c"
server_key: "8b66baadd136ad8a85c5f4ac4071ca0c"
server_key_iv: "b7e3dadbc5f8dfe796d315e8bab49150"
server_key_version: "encryptionkey@0"
I20260620 09:16:47.342314 4021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:47.363966 4021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:47.365881 4021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:47.366007 4021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:47.366214 4021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:47.366490 4021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:47.366521 4021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:47.366549 4021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:47.366572 4021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:47.371800 4021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44851
I20260620 09:16:47.371865 4149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44851 every 8 connection(s)
I20260620 09:16:47.372129 4021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:16:47.375109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 4021
I20260620 09:16:47.375206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:16:47.375376 392 external_mini_cluster.cc:1428] Setting key a14c9087fb1c87a0afefde866a5be026
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:47.381102 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:47.372829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43575 (local address 127.0.98.62:37219)
0620 09:16:47.372876 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:47.372879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:47.373043 (+ 164us) server_negotiation.cc:407] Connection header received
0620 09:16:47.373947 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:47.373948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:47.373959 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:47.374009 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:47.374596 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.375177 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.376376 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.376648 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.376718 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:47.378605 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:47.378607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:47.378607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:47.378614 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:47.379880 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:47.380268 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:47.380269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:47.380269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:47.380321 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:47.380525 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:47.380526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:47.380526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:47.380630 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:47.380695 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:47.380932 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 09:16:47.381015 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:16:47.381268 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:47.372761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37219 (local address 127.0.98.1:43575)
0620 09:16:47.372930 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:47.372970 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:47.373849 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:47.374085 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:47.374089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:47.374330 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:47.374510 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:47.374514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.375293 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.375301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:47.376301 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:47.376303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.376729 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.376729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:47.376829 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:47.377285 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:47.377293 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:47.378525 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:47.379953 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:47.379958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:47.379960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:47.380206 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:47.380373 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:47.380374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:47.380374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:47.380473 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:47.380685 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:47.380690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:47.380823 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 09:16:47.380908 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 09:16:47.381052 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260620 09:16:47.381691 4150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37219
I20260620 09:16:47.381770 4150 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:47.381927 4150 heartbeater.cc:511] Master 127.0.98.62:37219 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:16:47.382401 3959 ts_manager.cc:194] Registered new tserver with Master: 8a43bc49820743f18d817999e83a1734 (127.0.98.1:44851)
I20260620 09:16:47.382974 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:16:47.396029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37219
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:47.473260 4155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:47.473457 4155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:47.473479 4155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:47.474584 4155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:47.474615 4155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:47.474639 4155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:47.476109 4155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:47.476302 4155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:47.476464 4155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:16:47.481086 4155 server_base.cc:1034] running on GCE node
W20260620 09:16:47.481065 4161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:47.481071 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:47.481071 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:47.481536 4155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:47.481738 4155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:47.482867 4155 hybrid_clock.cc:648] HybridClock initialized: now 1781947007482855 us; error 30 us; skew 500 ppm
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:47.485154 4155 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:16:47.485533 4155 webserver.cc:460] Webserver started at http://127.0.98.2:42697/ using document root <none> and password file <none>
I20260620 09:16:47.485724 4155 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:47.485770 4155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:47.485890 4155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:47.486871 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "9b7a5e4668ce4d17ad56e5a8243111b6"
format_stamp: "Formatted at 2026-06-20 09:16:47 on dist-test-slave-rq8c"
server_key: "b739ba2a97c15d0c3e82bec0dea83ddd"
server_key_iv: "9f4949a7d2bb768a46bd72a6dd661dcf"
server_key_version: "encryptionkey@0"
I20260620 09:16:47.487179 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9b7a5e4668ce4d17ad56e5a8243111b6"
format_stamp: "Formatted at 2026-06-20 09:16:47 on dist-test-slave-rq8c"
server_key: "b739ba2a97c15d0c3e82bec0dea83ddd"
server_key_iv: "9f4949a7d2bb768a46bd72a6dd661dcf"
server_key_version: "encryptionkey@0"
I20260620 09:16:47.488413 4155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:47.489030 4170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:47.489166 4155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:47.489241 4155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "9b7a5e4668ce4d17ad56e5a8243111b6"
format_stamp: "Formatted at 2026-06-20 09:16:47 on dist-test-slave-rq8c"
server_key: "b739ba2a97c15d0c3e82bec0dea83ddd"
server_key_iv: "9f4949a7d2bb768a46bd72a6dd661dcf"
server_key_version: "encryptionkey@0"
I20260620 09:16:47.489302 4155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:47.498256 4155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:47.499918 4155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:47.500033 4155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:47.500216 4155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:47.500471 4155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:47.500501 4155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:47.500526 4155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:47.500551 4155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:47.505846 4155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40591
I20260620 09:16:47.505900 4283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40591 every 8 connection(s)
I20260620 09:16:47.506167 4155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:16:47.509910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 4155
I20260620 09:16:47.509999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:16:47.510143 392 external_mini_cluster.cc:1428] Setting key 9d139000bdeb772614a894eaf48217f7
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:47.515139 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:47.506784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57953 (local address 127.0.98.62:37219)
0620 09:16:47.506826 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:47.506828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:47.507032 (+ 204us) server_negotiation.cc:407] Connection header received
0620 09:16:47.507963 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:47.507964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:47.507975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:47.508027 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:47.508629 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.509107 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.510102 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.510308 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.510363 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:47.512614 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:47.512616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:47.512616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:47.512625 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:47.513943 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:47.514306 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:47.514307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:47.514308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:47.514367 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:47.514561 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:47.514561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:47.514562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:47.514701 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:47.514770 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:47.514972 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 09:16:47.515048 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:16:47.515204 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:47.506758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37219 (local address 127.0.98.2:57953)
0620 09:16:47.506920 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:47.506953 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:47.507871 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:47.508106 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:47.508111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:47.508364 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:47.508543 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:47.508547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.509182 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.509183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:47.510002 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:47.510004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.510376 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.510377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:47.510510 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:47.511008 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:47.511016 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:47.512526 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:47.514024 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:47.514029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:47.514031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:47.514241 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:47.514424 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:47.514424 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:47.514425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:47.514507 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:47.514756 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:47.514758 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:47.514863 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 09:16:47.514936 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:16:47.515047 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260620 09:16:47.515689 4284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37219
I20260620 09:16:47.515784 4284 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:47.515960 4284 heartbeater.cc:511] Master 127.0.98.62:37219 requested a full tablet report, sending...
I20260620 09:16:47.516345 3959 ts_manager.cc:194] Registered new tserver with Master: 9b7a5e4668ce4d17ad56e5a8243111b6 (127.0.98.2:40591)
I20260620 09:16:47.516706 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57953
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:16:47.531836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37219
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:47.609347 4289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:47.609547 4289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:47.609570 4289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:47.610706 4289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:47.610747 4289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:47.610780 4289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:47.612483 4289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:47.612720 4289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:47.612923 4289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:47.617655 4295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:47.617648 4297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:47.617645 4294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:47.618036 4289 server_base.cc:1034] running on GCE node
I20260620 09:16:47.618278 4289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:47.618508 4289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:47.619654 4289 hybrid_clock.cc:648] HybridClock initialized: now 1781947007619639 us; error 36 us; skew 500 ppm
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:47.622043 4289 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:16:47.622434 4289 webserver.cc:460] Webserver started at http://127.0.98.3:34831/ using document root <none> and password file <none>
I20260620 09:16:47.622639 4289 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:47.622681 4289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:47.622805 4289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:47.623869 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "1f93cf718fb9480f8306844e60964b56"
format_stamp: "Formatted at 2026-06-20 09:16:47 on dist-test-slave-rq8c"
server_key: "fec896b71fcec902b968ccfce7cf2555"
server_key_iv: "91eb1a28b448024ce756700b6ba10a6c"
server_key_version: "encryptionkey@0"
I20260620 09:16:47.624186 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1f93cf718fb9480f8306844e60964b56"
format_stamp: "Formatted at 2026-06-20 09:16:47 on dist-test-slave-rq8c"
server_key: "fec896b71fcec902b968ccfce7cf2555"
server_key_iv: "91eb1a28b448024ce756700b6ba10a6c"
server_key_version: "encryptionkey@0"
I20260620 09:16:47.625486 4289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:47.626138 4304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:47.626307 4289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:47.626382 4289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "1f93cf718fb9480f8306844e60964b56"
format_stamp: "Formatted at 2026-06-20 09:16:47 on dist-test-slave-rq8c"
server_key: "fec896b71fcec902b968ccfce7cf2555"
server_key_iv: "91eb1a28b448024ce756700b6ba10a6c"
server_key_version: "encryptionkey@0"
I20260620 09:16:47.626448 4289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:47.649634 4289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:47.651577 4289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:47.651736 4289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:47.651958 4289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:47.652256 4289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:47.652290 4289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:47.652312 4289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:47.652335 4289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:47.657480 4289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39477
I20260620 09:16:47.657562 4417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39477 every 8 connection(s)
I20260620 09:16:47.657807 4289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:47.666332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 4289
I20260620 09:16:47.666414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:16:47.666601 392 external_mini_cluster.cc:1428] Setting key d4e2bc9d35e4e3289342e6d6cde50f7f
I20260620 09:16:47.666652 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:47.658456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35149 (local address 127.0.98.62:37219)
0620 09:16:47.658526 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:47.658528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:47.658721 (+ 193us) server_negotiation.cc:407] Connection header received
0620 09:16:47.659581 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:47.659582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:47.659594 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:47.659690 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:47.660250 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.660737 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.661757 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.661950 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.662012 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:47.663937 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:47.663939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:47.663939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:47.663946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:47.665384 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:47.665773 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:47.665775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:47.665775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:47.665833 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:47.666021 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:47.666021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:47.666021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:47.666120 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:47.666173 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:47.666520 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 09:16:47.666584 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:16:47.666860 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:47.658443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37219 (local address 127.0.98.3:35149)
0620 09:16:47.658611 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:47.658645 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:47.659494 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:47.659748 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:47.659751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:47.659992 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:47.660171 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:47.660176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.660876 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.660877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:47.661670 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:47.661672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.662018 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.662019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:47.662113 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:47.662522 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:47.662530 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:47.663848 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:47.665470 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:47.665476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:47.665478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:47.665693 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:47.665889 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:47.665890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:47.665890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:47.665977 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:47.666197 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:47.666205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:47.666390 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 09:16:47.666506 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 09:16:47.666665 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260620 09:16:47.667327 4418 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37219
I20260620 09:16:47.667423 4418 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:47.667593 4418 heartbeater.cc:511] Master 127.0.98.62:37219 requested a full tablet report, sending...
I20260620 09:16:47.667940 3959 ts_manager.cc:194] Registered new tserver with Master: 1f93cf718fb9480f8306844e60964b56 (127.0.98.3:39477)
I20260620 09:16:47.668280 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35149
I20260620 09:16:47.678193 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:16:47.699110 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:47.692186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59922 (local address 127.0.98.1:44851)
0620 09:16:47.692320 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:47.692325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:47.692336 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:16:47.692360 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:47.692362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:47.692377 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:47.692434 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:47.692994 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.693772 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.694563 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.694880 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.694938 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:47.696891 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:47.696899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:47.696899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:47.696906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:47.698111 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:47.698397 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:47.698401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:47.698401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:47.698451 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:47.698608 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:47.698609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:47.698609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:47.698734 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:47.698864 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:47.698930 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 09:16:47.699006 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:47.705852 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:47.700263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.0.98.62:37219)
0620 09:16:47.700305 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:47.700307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:47.700315 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:47.700353 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:47.700354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:47.700362 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:47.700403 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:47.700815 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.701309 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.701937 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:47.702133 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:47.702186 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:47.704048 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:47.704049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:47.704049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:47.704055 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:47.705064 (+ 1009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:47.705320 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:47.705321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:47.705321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:47.705362 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:47.705480 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:47.705480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:47.705480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:47.705599 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:47.705661 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:47.705742 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 09:16:47.705791 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 09:16:47.706046 3959 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57004
I20260620 09:16:47.706346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 4021
I20260620 09:16:47.711501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 4155
I20260620 09:16:47.715536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 4289
I20260620 09:16:47.719069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 3927
2026-06-20T09:16:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1756 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[4430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[4430](info): set up 2 sockets
Jun 20 09:16:47 dist-test-slave-rq8c krb5kdc[4430](info): commencing operation
krb5kdc: starting...
W20260620 09:16:48.663286 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:16:48 dist-test-slave-rq8c krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:16:48.768446 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42697
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:48.846274 4443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:48.846477 4443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:48.846505 4443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:48.847645 4443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:48.847681 4443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:48.847700 4443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:48.847716 4443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:48.847731 4443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:48.849200 4443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:48.849428 4443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:48.849628 4443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:48.854071 4451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:48.854135 4449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:48.854137 4448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:48.854293 4443 server_base.cc:1034] running on GCE node
I20260620 09:16:48.854494 4443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:48.854755 4443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:48.855895 4443 hybrid_clock.cc:648] HybridClock initialized: now 1781947008855884 us; error 29 us; skew 500 ppm
Jun 20 09:16:48 dist-test-slave-rq8c krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:48.858168 4443 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:16:48.858467 4443 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:48.858516 4443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:48.858652 4443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:48.859688 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "a51b7dd75fe5493681512ff3f6e57a37"
format_stamp: "Formatted at 2026-06-20 09:16:48 on dist-test-slave-rq8c"
server_key: "337cf6acf384d7cbe74198792cc9bc27"
server_key_iv: "063f4b78a7e9a7a47be88591dcdff8c1"
server_key_version: "encryptionkey@0"
I20260620 09:16:48.860003 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "a51b7dd75fe5493681512ff3f6e57a37"
format_stamp: "Formatted at 2026-06-20 09:16:48 on dist-test-slave-rq8c"
server_key: "337cf6acf384d7cbe74198792cc9bc27"
server_key_iv: "063f4b78a7e9a7a47be88591dcdff8c1"
server_key_version: "encryptionkey@0"
I20260620 09:16:48.861140 4443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:48.861786 4457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:48.861937 4443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:48.862015 4443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "a51b7dd75fe5493681512ff3f6e57a37"
format_stamp: "Formatted at 2026-06-20 09:16:48 on dist-test-slave-rq8c"
server_key: "337cf6acf384d7cbe74198792cc9bc27"
server_key_iv: "063f4b78a7e9a7a47be88591dcdff8c1"
server_key_version: "encryptionkey@0"
I20260620 09:16:48.862074 4443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:48.871166 4443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:48.872754 4443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:48.872870 4443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:48.876333 4443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42697
I20260620 09:16:48.876355 4509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42697 every 8 connection(s)
I20260620 09:16:48.876586 4443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:48.877413 4510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:48.879532 4510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37: Bootstrap starting.
I20260620 09:16:48.880079 4510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:48.880343 4510 log.cc:824] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:48.880903 4510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37: No bootstrap required, opened a new log
I20260620 09:16:48.882443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 4443
I20260620 09:16:48.882548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:48.882436 4510 raft_consensus.cc:348] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51b7dd75fe5493681512ff3f6e57a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42697 } }
I20260620 09:16:48.882591 4510 raft_consensus.cc:374] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:48.882632 4510 raft_consensus.cc:729] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51b7dd75fe5493681512ff3f6e57a37, State: Initialized, Role: FOLLOWER
I20260620 09:16:48.882720 392 external_mini_cluster.cc:1428] Setting key 1956dc86d9aefde1cd6bb25306e3960d
I20260620 09:16:48.882776 4510 consensus_queue.cc:260] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51b7dd75fe5493681512ff3f6e57a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42697 } }
I20260620 09:16:48.882854 4510 raft_consensus.cc:388] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:48.882890 4510 raft_consensus.cc:482] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:48.882934 4510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:48.883570 4510 raft_consensus.cc:504] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51b7dd75fe5493681512ff3f6e57a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42697 } }
I20260620 09:16:48.883695 4510 leader_election.cc:302] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a51b7dd75fe5493681512ff3f6e57a37; no voters:
I20260620 09:16:48.883834 4510 leader_election.cc:288] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:48.883869 4515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:48.883972 4515 raft_consensus.cc:686] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [term 1 LEADER]: Becoming Leader. State: Replica: a51b7dd75fe5493681512ff3f6e57a37, State: Running, Role: LEADER
I20260620 09:16:48.884047 4515 consensus_queue.cc:237] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51b7dd75fe5493681512ff3f6e57a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42697 } }
I20260620 09:16:48.884073 4510 sys_catalog.cc:564] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:48.884366 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a51b7dd75fe5493681512ff3f6e57a37. Latest consensus state: current_term: 1 leader_uuid: "a51b7dd75fe5493681512ff3f6e57a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51b7dd75fe5493681512ff3f6e57a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42697 } } }
I20260620 09:16:48.884438 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:48.884377 4517 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a51b7dd75fe5493681512ff3f6e57a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51b7dd75fe5493681512ff3f6e57a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42697 } } }
I20260620 09:16:48.884694 4517 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:48.885052 4520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:48.885519 4520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:48.887539 4520 catalog_manager.cc:1347] Generated new cluster ID: 036388fe424840a5b9a662d994aeea7b
I20260620 09:16:48.887594 4520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:16:48 dist-test-slave-rq8c krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:48.892812 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:48.883149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.0.98.62:42697)
0620 09:16:48.883723 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:48.883731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:48.883753 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:16:48.883857 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:48.883864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:48.883918 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:48.884773 (+ 855us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:48.884863 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:48.886405 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:48.887132 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:48.887737 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:48.887835 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:48.889985 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:48.889988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:48.889989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:48.889998 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:48.891410 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:48.891738 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:48.891740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:48.891741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:48.891816 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:48.891995 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:48.891996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:48.891996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:48.892219 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:48.892354 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:48.892442 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 09:16:48.892661 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":439,"threads_started":1}
I20260620 09:16:48.897202 4520 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:48.897866 4520 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:48.905165 4520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37: Generated new TSK 0
I20260620 09:16:48.905321 4520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:16:48.934861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42697
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:49.012010 4536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:49.012217 4536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:49.012248 4536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:49.013387 4536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:49.013425 4536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:49.013461 4536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:49.014947 4536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:49.015175 4536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:49.015372 4536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:49.020151 4541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:49.020210 4536 server_base.cc:1034] running on GCE node
W20260620 09:16:49.020124 4544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:49.020136 4542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:49.020473 4536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:49.020668 4536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:49.021816 4536 hybrid_clock.cc:648] HybridClock initialized: now 1781947009021799 us; error 30 us; skew 500 ppm
Jun 20 09:16:49 dist-test-slave-rq8c krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:49.024085 4536 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:16:49.024381 4536 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:49.024441 4536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:49.024564 4536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:49.025569 4536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f06c99ebe114f45bcbcd79c560bf185"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "d185829db76ec30e89abe63d9cef59e3"
server_key_iv: "17d774b69ef0018828587865f139bfe2"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.025898 4536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f06c99ebe114f45bcbcd79c560bf185"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "d185829db76ec30e89abe63d9cef59e3"
server_key_iv: "17d774b69ef0018828587865f139bfe2"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.027104 4536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:49.027915 4550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.028080 4536 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:49.028163 4536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "2f06c99ebe114f45bcbcd79c560bf185"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "d185829db76ec30e89abe63d9cef59e3"
server_key_iv: "17d774b69ef0018828587865f139bfe2"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.028231 4536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:49.060175 4536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:49.062054 4536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:49.062183 4536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:49.062387 4536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:49.062691 4536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:49.062731 4536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.062759 4536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:49.062776 4536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.068185 4536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46699
I20260620 09:16:49.068231 4663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46699 every 8 connection(s)
I20260620 09:16:49.068465 4536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:16:49.069587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 4536
I20260620 09:16:49.069695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:16:49.069881 392 external_mini_cluster.cc:1428] Setting key fbafa8b79d44e924a381cc17b6c573c9
Jun 20 09:16:49 dist-test-slave-rq8c krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:16:49.077987 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.069106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50189 (local address 127.0.98.62:42697)
0620 09:16:49.069169 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.069171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.069351 (+ 180us) server_negotiation.cc:407] Connection header received
0620 09:16:49.069986 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.069988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.070003 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.070062 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:49.071151 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.071703 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.072727 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.072994 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.073057 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.075324 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:49.075326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:49.075327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:49.075333 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:49.076642 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:49.077078 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:49.077079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:49.077079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:49.077135 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:49.077375 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:49.077375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:49.077375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:49.077480 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:49.077541 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.077857 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.077922 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:16:49.078182 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.069075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42697 (local address 127.0.98.1:50189)
0620 09:16:49.069231 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.069272 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.069878 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.070148 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.070151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.070398 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:49.071036 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.071044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.071794 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.071795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.072642 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.072644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.073111 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.073112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.073223 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.073693 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:49.073701 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:49.075237 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:49.076717 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:49.076724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:49.076726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:49.076989 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:49.077190 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:49.077191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:49.077191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:49.077319 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:49.077540 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:49.077545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:49.077707 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.077841 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.077982 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260620 09:16:49.078692 4664 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42697
I20260620 09:16:49.078783 4664 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:49.078929 4664 heartbeater.cc:511] Master 127.0.98.62:42697 requested a full tablet report, sending...
I20260620 09:16:49.079411 4474 ts_manager.cc:194] Registered new tserver with Master: 2f06c99ebe114f45bcbcd79c560bf185 (127.0.98.1:46699)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:16:49.080251 4474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:16:49.092254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42697
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:49.172245 4669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:49.172441 4669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:49.172462 4669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:49.173563 4669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:49.173596 4669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:49.173621 4669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:49.175035 4669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:49.175225 4669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:49.175401 4669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:49.180099 4675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:49.180107 4677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:49.180147 4674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:49.180299 4669 server_base.cc:1034] running on GCE node
I20260620 09:16:49.180485 4669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:49.180714 4669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:49.181865 4669 hybrid_clock.cc:648] HybridClock initialized: now 1781947009181843 us; error 35 us; skew 500 ppm
Jun 20 09:16:49 dist-test-slave-rq8c krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:49.188855 4669 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:16:49.189184 4669 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:49.189235 4669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:49.189354 4669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:49.190369 4669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "78a0c2a8e49b4e39ab9605172a63a9f4"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "7df9b98d393501a4a56541da44da5a3c"
server_key_iv: "7470991cf2e12d8b382836f1f79965fc"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.190685 4669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78a0c2a8e49b4e39ab9605172a63a9f4"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "7df9b98d393501a4a56541da44da5a3c"
server_key_iv: "7470991cf2e12d8b382836f1f79965fc"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.191916 4669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:16:49.192610 4683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.192766 4669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:49.192847 4669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "78a0c2a8e49b4e39ab9605172a63a9f4"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "7df9b98d393501a4a56541da44da5a3c"
server_key_iv: "7470991cf2e12d8b382836f1f79965fc"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.192903 4669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:49.208500 4669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:49.210595 4669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:49.210727 4669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:49.210928 4669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:49.211227 4669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:49.211259 4669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.211287 4669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:49.211311 4669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.216657 4669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33517
I20260620 09:16:49.216692 4796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33517 every 8 connection(s)
I20260620 09:16:49.216948 4669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:16:49.217355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 4669
I20260620 09:16:49.217449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:16:49.217641 392 external_mini_cluster.cc:1428] Setting key 57d393a7131f2b8e8f4f6bf06ef07016
Jun 20 09:16:49 dist-test-slave-rq8c krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:16:49.226464 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.217619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34847 (local address 127.0.98.62:42697)
0620 09:16:49.217680 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.217682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.217916 (+ 234us) server_negotiation.cc:407] Connection header received
0620 09:16:49.218425 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.218426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.218438 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.218490 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:49.219422 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.219974 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.221003 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.221318 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.221393 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.223578 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:49.223581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:49.223581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:49.223589 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:49.225161 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:49.225591 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:49.225592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:49.225593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:49.225656 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:49.225876 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:49.225877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:49.225878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:49.226020 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:49.226093 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.226317 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.226388 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:16:49.226590 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.217607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42697 (local address 127.0.98.2:34847)
0620 09:16:49.217802 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.217842 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.218337 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.218545 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.218548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.218792 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:49.219333 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.219340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.220050 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.220051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.220916 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.220919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.221404 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.221405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.221515 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.222019 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:49.222027 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:49.223486 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:49.225256 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:49.225263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:49.225265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:49.225522 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:49.225719 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:49.225719 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:49.225720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:49.225806 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:49.226080 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:49.226102 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:49.226216 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.226298 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.226412 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260620 09:16:49.227124 4797 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:16:49.227214 4797 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:49.227345 4797 heartbeater.cc:511] Master 127.0.98.62:42697 requested a full tablet report, sending...
I20260620 09:16:49.227723 4474 ts_manager.cc:194] Registered new tserver with Master: 78a0c2a8e49b4e39ab9605172a63a9f4 (127.0.98.2:33517)
I20260620 09:16:49.228125 4474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:16:49.239735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42697
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:49.316072 4802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:49.316275 4802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:49.316306 4802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:49.317427 4802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:49.317466 4802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:49.317499 4802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:49.318934 4802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:49.319144 4802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:49.319339 4802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:16:49.323899 4802 server_base.cc:1034] running on GCE node
W20260620 09:16:49.323870 4807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:49.323870 4810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:49.323913 4808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:49.324301 4802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:49.324538 4802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:49.325671 4802 hybrid_clock.cc:648] HybridClock initialized: now 1781947009325657 us; error 30 us; skew 500 ppm
Jun 20 09:16:49 dist-test-slave-rq8c krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:49.328205 4802 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:16:49.328491 4802 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:49.328537 4802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:49.328652 4802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:49.329600 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "1095055f896d416daeefb41c7e1af5d6"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "75bf120b95f7f51c50a9a8960fc44ff2"
server_key_iv: "0b3f04a7e904be191cfb7848168c0b19"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.329905 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1095055f896d416daeefb41c7e1af5d6"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "75bf120b95f7f51c50a9a8960fc44ff2"
server_key_iv: "0b3f04a7e904be191cfb7848168c0b19"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.331002 4802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:49.331655 4816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.331805 4802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:49.331880 4802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "1095055f896d416daeefb41c7e1af5d6"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "75bf120b95f7f51c50a9a8960fc44ff2"
server_key_iv: "0b3f04a7e904be191cfb7848168c0b19"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.331943 4802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:49.338660 4802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:49.340565 4802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:49.340683 4802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:49.340862 4802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:49.341131 4802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:49.341162 4802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.341188 4802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:49.341210 4802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.346300 4802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34085
I20260620 09:16:49.346355 4929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34085 every 8 connection(s)
I20260620 09:16:49.346555 4802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:16:49 dist-test-slave-rq8c krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:49.353471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 4802
I20260620 09:16:49.353562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:16:49.353756 392 external_mini_cluster.cc:1428] Setting key 5f953821bfdddf367a8382bc25ee65d8
I20260620 09:16:49.354786 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.347147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41895 (local address 127.0.98.62:42697)
0620 09:16:49.347217 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.347219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.347357 (+ 138us) server_negotiation.cc:407] Connection header received
0620 09:16:49.347868 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.347869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.347878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.347927 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:49.348752 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.349248 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.350303 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.350501 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.350579 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.352434 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:49.352436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:49.352436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:49.352443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:49.353490 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:49.353911 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:49.353912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:49.353913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:49.353977 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:49.354218 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:49.354218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:49.354219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:49.354346 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:49.354405 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.354667 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.354725 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:16:49.354938 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.347119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42697 (local address 127.0.98.3:41895)
0620 09:16:49.347249 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.347282 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.347781 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.347978 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.347981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.348215 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:49.348651 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.348656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.349316 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.349317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.350208 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.350212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.350586 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.350587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.350689 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.351088 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:49.351096 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:49.352352 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:49.353566 (+ 1214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:49.353581 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:49.353583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:49.353846 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:49.354035 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:49.354035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:49.354035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:49.354160 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:49.354401 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:49.354405 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:49.354535 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.354642 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.354753 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260620 09:16:49.355320 4930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42697
I20260620 09:16:49.355399 4930 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:49.355530 4930 heartbeater.cc:511] Master 127.0.98.62:42697 requested a full tablet report, sending...
I20260620 09:16:49.355844 4474 ts_manager.cc:194] Registered new tserver with Master: 1095055f896d416daeefb41c7e1af5d6 (127.0.98.3:34085)
I20260620 09:16:49.356220 4474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41895
I20260620 09:16:49.365332 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:16:49.371904 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.366546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.0.98.62:42697)
0620 09:16:49.366607 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.366609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.366701 (+ 92us) server_negotiation.cc:407] Connection header received
0620 09:16:49.366766 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.366767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.366776 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.366825 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:49.367383 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.367903 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.368597 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.368782 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.368834 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.369909 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:49.369910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:49.369911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:49.369918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:49.370877 (+ 959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:49.371169 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:49.371170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:49.371170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:49.371218 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:49.371371 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:49.371372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:49.371372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:49.371506 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:49.371591 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.371739 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.371812 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:16:49.373308 4474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:49.374802 4474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:49.380961 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.377260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.0.98.3:34085)
0620 09:16:49.377500 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.377505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.377514 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:16:49.377618 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.377620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.377636 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.377771 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:49.377880 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.378541 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.379977 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.380650 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.380710 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.380760 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.380823 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.380867 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
I20260620 09:16:49.380988 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.376798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46699 (local address 127.0.0.1:34702)
0620 09:16:49.376920 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.376961 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.377026 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.377378 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.377381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.377384 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:49.377456 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.377457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.378834 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.378834 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.379932 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.379934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.380579 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.380581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.380656 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.380669 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.380820 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.380889 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":26,"thread_start_us":60,"threads_started":1}
I20260620 09:16:49.381045 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.377056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.0.98.2:33517)
0620 09:16:49.377364 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.377369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.377378 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:16:49.377411 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.377413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.377428 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.377500 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:49.377718 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.378757 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.379847 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.380549 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.380636 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.380694 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.380930 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.380973 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":87,"threads_started":1}
I20260620 09:16:49.381191 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.376848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33517 (local address 127.0.0.1:58356)
0620 09:16:49.377225 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.377234 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.377269 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.377586 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.377586 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.377589 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:49.377637 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.377638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.378854 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.378855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.379776 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.379778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.380728 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.380729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.380790 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.380795 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.380840 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.380892 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":6,"thread_start_us":56,"threads_started":1}
I20260620 09:16:49.381335 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.377066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.0.98.1:46699)
0620 09:16:49.377204 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.377209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.377218 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:16:49.377259 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.377261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.377276 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.377360 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:49.377539 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.378739 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.380012 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.380501 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.380568 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.380643 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.381206 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.381258 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260620 09:16:49.380995 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.376848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34085 (local address 127.0.0.1:42056)
0620 09:16:49.377063 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.377073 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.377131 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.377782 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.377783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.377786 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:49.377827 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.377828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.378638 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.378639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.379894 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.379896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.380710 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.380711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.380763 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.380765 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.380811 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.380887 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260620 09:16:49.381916 4598 tablet_service.cc:1467] Processing CreateTablet for tablet 553d4497fccf4a3ca92301fa4bd5229b (DEFAULT_TABLE table=test-table [id=89b6ba29f43a4e1c830ff303739381c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:49.381906 4731 tablet_service.cc:1467] Processing CreateTablet for tablet 553d4497fccf4a3ca92301fa4bd5229b (DEFAULT_TABLE table=test-table [id=89b6ba29f43a4e1c830ff303739381c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:49.382272 4598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553d4497fccf4a3ca92301fa4bd5229b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:49.382272 4731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553d4497fccf4a3ca92301fa4bd5229b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:49.384313 4864 tablet_service.cc:1467] Processing CreateTablet for tablet 553d4497fccf4a3ca92301fa4bd5229b (DEFAULT_TABLE table=test-table [id=89b6ba29f43a4e1c830ff303739381c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:49.384629 4864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553d4497fccf4a3ca92301fa4bd5229b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:49.385682 4944 tablet_bootstrap.cc:492] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4: Bootstrap starting.
I20260620 09:16:49.386008 4945 tablet_bootstrap.cc:492] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185: Bootstrap starting.
I20260620 09:16:49.386497 4945 tablet_bootstrap.cc:654] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:49.386696 4944 tablet_bootstrap.cc:654] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:49.386919 4945 log.cc:824] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:49.387365 4944 log.cc:824] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:49.387539 4945 tablet_bootstrap.cc:492] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185: No bootstrap required, opened a new log
I20260620 09:16:49.387598 4945 ts_tablet_manager.cc:1397] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:49.388018 4947 tablet_bootstrap.cc:492] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6: Bootstrap starting.
I20260620 09:16:49.388619 4947 tablet_bootstrap.cc:654] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:49.389214 4947 log.cc:824] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:49.389549 4945 raft_consensus.cc:348] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "1095055f896d416daeefb41c7e1af5d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34085 } } peers { permanent_uuid: "2f06c99ebe114f45bcbcd79c560bf185" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } }
I20260620 09:16:49.389714 4945 raft_consensus.cc:374] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:49.389756 4945 raft_consensus.cc:729] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f06c99ebe114f45bcbcd79c560bf185, State: Initialized, Role: FOLLOWER
I20260620 09:16:49.389875 4945 consensus_queue.cc:260] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "1095055f896d416daeefb41c7e1af5d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34085 } } peers { permanent_uuid: "2f06c99ebe114f45bcbcd79c560bf185" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } }
I20260620 09:16:49.389925 4947 tablet_bootstrap.cc:492] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6: No bootstrap required, opened a new log
I20260620 09:16:49.389977 4947 ts_tablet_manager.cc:1397] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:49.390167 4945 ts_tablet_manager.cc:1428] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:16:49.390193 4664 heartbeater.cc:503] Master 127.0.98.62:42697 was elected leader, sending a full tablet report...
I20260620 09:16:49.390825 4944 tablet_bootstrap.cc:492] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4: No bootstrap required, opened a new log
I20260620 09:16:49.390893 4944 ts_tablet_manager.cc:1397] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 09:16:49.391705 4947 raft_consensus.cc:348] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "1095055f896d416daeefb41c7e1af5d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34085 } } peers { permanent_uuid: "2f06c99ebe114f45bcbcd79c560bf185" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } }
I20260620 09:16:49.391827 4947 raft_consensus.cc:374] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:49.391847 4947 raft_consensus.cc:729] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1095055f896d416daeefb41c7e1af5d6, State: Initialized, Role: FOLLOWER
I20260620 09:16:49.391942 4947 consensus_queue.cc:260] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "1095055f896d416daeefb41c7e1af5d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34085 } } peers { permanent_uuid: "2f06c99ebe114f45bcbcd79c560bf185" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } }
I20260620 09:16:49.392182 4947 ts_tablet_manager.cc:1428] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:16:49.392221 4930 heartbeater.cc:503] Master 127.0.98.62:42697 was elected leader, sending a full tablet report...
I20260620 09:16:49.392648 4944 raft_consensus.cc:348] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "1095055f896d416daeefb41c7e1af5d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34085 } } peers { permanent_uuid: "2f06c99ebe114f45bcbcd79c560bf185" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } }
I20260620 09:16:49.392802 4944 raft_consensus.cc:374] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:49.392843 4944 raft_consensus.cc:729] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a0c2a8e49b4e39ab9605172a63a9f4, State: Initialized, Role: FOLLOWER
I20260620 09:16:49.392941 4944 consensus_queue.cc:260] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "1095055f896d416daeefb41c7e1af5d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34085 } } peers { permanent_uuid: "2f06c99ebe114f45bcbcd79c560bf185" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } }
I20260620 09:16:49.393167 4944 ts_tablet_manager.cc:1428] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:16:49.393199 4797 heartbeater.cc:503] Master 127.0.98.62:42697 was elected leader, sending a full tablet report...
I20260620 09:16:49.432206 4950 raft_consensus.cc:482] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:49.432324 4950 raft_consensus.cc:504] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "1095055f896d416daeefb41c7e1af5d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34085 } } peers { permanent_uuid: "2f06c99ebe114f45bcbcd79c560bf185" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } }
I20260620 09:16:49.432641 4950 leader_election.cc:288] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78a0c2a8e49b4e39ab9605172a63a9f4 (127.0.98.2:33517), 1095055f896d416daeefb41c7e1af5d6 (127.0.98.3:34085)
I20260620 09:16:49.435491 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.432965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46965 (local address 127.0.98.2:33517)
0620 09:16:49.433033 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.433036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.433121 (+ 85us) server_negotiation.cc:407] Connection header received
0620 09:16:49.433183 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.433183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.433195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.433249 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:49.433391 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.433859 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.434800 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.435183 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.435234 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.435242 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.435385 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.435418 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:16:49.435509 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.432846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33517 (local address 127.0.98.1:46965)
0620 09:16:49.433052 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.433063 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.433135 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.433257 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.433258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.433260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:49.433327 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.433328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.433930 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.433931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.434718 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.434720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.435253 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.435254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.435321 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.435325 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.435374 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.435412 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":22,"thread_start_us":109,"threads_started":1}
I20260620 09:16:49.435539 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.432944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36077 (local address 127.0.98.3:34085)
0620 09:16:49.433032 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.433034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.433045 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:16:49.433073 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.433074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.433086 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.433142 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:49.433267 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.433767 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.434753 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.435244 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.435308 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.435318 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.435452 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.435484 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:16:49.435760 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.432846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34085 (local address 127.0.98.1:36077)
0620 09:16:49.432905 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.432916 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.432964 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.433152 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.433153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.433155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:49.433209 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.433211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.433832 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.433833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.434676 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:49.434678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.435315 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.435316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:49.435386 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.435390 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.435439 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.435476 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":1}
I20260620 09:16:49.435928 4751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d4497fccf4a3ca92301fa4bd5229b" candidate_uuid: "2f06c99ebe114f45bcbcd79c560bf185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4" is_pre_election: true
I20260620 09:16:49.436086 4751 raft_consensus.cc:2454] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f06c99ebe114f45bcbcd79c560bf185 in term 0.
I20260620 09:16:49.436049 4884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d4497fccf4a3ca92301fa4bd5229b" candidate_uuid: "2f06c99ebe114f45bcbcd79c560bf185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1095055f896d416daeefb41c7e1af5d6" is_pre_election: true
I20260620 09:16:49.436169 4884 raft_consensus.cc:2454] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f06c99ebe114f45bcbcd79c560bf185 in term 0.
I20260620 09:16:49.436295 4551 leader_election.cc:302] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f06c99ebe114f45bcbcd79c560bf185, 78a0c2a8e49b4e39ab9605172a63a9f4; no voters:
I20260620 09:16:49.436441 4950 raft_consensus.cc:2781] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:49.436522 4950 raft_consensus.cc:482] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:49.436538 4950 raft_consensus.cc:3037] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:49.437314 4950 raft_consensus.cc:504] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "1095055f896d416daeefb41c7e1af5d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34085 } } peers { permanent_uuid: "2f06c99ebe114f45bcbcd79c560bf185" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } }
I20260620 09:16:49.437433 4950 leader_election.cc:288] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [CANDIDATE]: Term 1 election: Requested vote from peers 78a0c2a8e49b4e39ab9605172a63a9f4 (127.0.98.2:33517), 1095055f896d416daeefb41c7e1af5d6 (127.0.98.3:34085)
I20260620 09:16:49.437605 4884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d4497fccf4a3ca92301fa4bd5229b" candidate_uuid: "2f06c99ebe114f45bcbcd79c560bf185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1095055f896d416daeefb41c7e1af5d6"
I20260620 09:16:49.437628 4751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553d4497fccf4a3ca92301fa4bd5229b" candidate_uuid: "2f06c99ebe114f45bcbcd79c560bf185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4"
I20260620 09:16:49.437704 4751 raft_consensus.cc:3037] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:49.437701 4884 raft_consensus.cc:3037] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:49.438556 4884 raft_consensus.cc:2454] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f06c99ebe114f45bcbcd79c560bf185 in term 1.
I20260620 09:16:49.438560 4751 raft_consensus.cc:2454] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f06c99ebe114f45bcbcd79c560bf185 in term 1.
I20260620 09:16:49.438728 4551 leader_election.cc:302] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f06c99ebe114f45bcbcd79c560bf185, 78a0c2a8e49b4e39ab9605172a63a9f4; no voters:
I20260620 09:16:49.438819 4950 raft_consensus.cc:2781] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:49.438961 4950 raft_consensus.cc:686] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [term 1 LEADER]: Becoming Leader. State: Replica: 2f06c99ebe114f45bcbcd79c560bf185, State: Running, Role: LEADER
I20260620 09:16:49.439041 4950 consensus_queue.cc:237] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "1095055f896d416daeefb41c7e1af5d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34085 } } peers { permanent_uuid: "2f06c99ebe114f45bcbcd79c560bf185" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } }
I20260620 09:16:49.439947 4474 catalog_manager.cc:5495] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f06c99ebe114f45bcbcd79c560bf185 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2f06c99ebe114f45bcbcd79c560bf185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1095055f896d416daeefb41c7e1af5d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f06c99ebe114f45bcbcd79c560bf185" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46699 } health_report { overall_health: HEALTHY } } }
I20260620 09:16:49.455855 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.453628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.0.98.1:46699)
0620 09:16:49.453680 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.453682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.453690 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:49.453720 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.453721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.453738 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.453800 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:49.453891 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.454343 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.455068 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:49.455296 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:49.455357 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:49.455491 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:49.455594 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:49.455728 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.455758 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.455782 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:16:49.457485 4751 raft_consensus.cc:1261] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4 [term 1 FOLLOWER]: Refusing update from remote peer 2f06c99ebe114f45bcbcd79c560bf185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:49.457485 4884 raft_consensus.cc:1261] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6 [term 1 FOLLOWER]: Refusing update from remote peer 2f06c99ebe114f45bcbcd79c560bf185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:49.457830 4950 consensus_queue.cc:1035] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1095055f896d416daeefb41c7e1af5d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:49.457964 4954 consensus_queue.cc:1035] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a0c2a8e49b4e39ab9605172a63a9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:49.460343 4959 mvcc.cc:205] Tried to move back new op lower bound from 7298854950735495168 to 7298854950663110656. Current Snapshot: MvccSnapshot[applied={T|T < 7298854950735495168}]
I20260620 09:16:49.460920 4958 mvcc.cc:205] Tried to move back new op lower bound from 7298854950735495168 to 7298854950663110656. Current Snapshot: MvccSnapshot[applied={T|T < 7298854950735495168}]
I20260620 09:16:49.462133 4473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:49.462212 4473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:49.462877 4473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a51b7dd75fe5493681512ff3f6e57a37: Sending DeleteTablet for 3 replicas of tablet 553d4497fccf4a3ca92301fa4bd5229b
I20260620 09:16:49.463173 4598 tablet_service.cc:1514] Processing DeleteTablet for tablet 553d4497fccf4a3ca92301fa4bd5229b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34702
I20260620 09:16:49.463184 4864 tablet_service.cc:1514] Processing DeleteTablet for tablet 553d4497fccf4a3ca92301fa4bd5229b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42056
I20260620 09:16:49.463213 4731 tablet_service.cc:1514] Processing DeleteTablet for tablet 553d4497fccf4a3ca92301fa4bd5229b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58356
I20260620 09:16:49.463366 4967 tablet_replica.cc:330] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185: stopping tablet replica
I20260620 09:16:49.463457 4967 raft_consensus.cc:2229] T 553d4497fccf4a3ca92301fa4bd5229b P 2f06c99ebe114f45bcbcd79c560bf185 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:49.463580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 4536
I20260620 09:16:49.464243 4969 tablet_replica.cc:330] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4: stopping tablet replica
I20260620 09:16:49.464344 4969 raft_consensus.cc:2229] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:49.464442 4969 raft_consensus.cc:2258] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:49.464593 4968 tablet_replica.cc:330] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6: stopping tablet replica
I20260620 09:16:49.464681 4968 raft_consensus.cc:2229] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:49.464768 4968 raft_consensus.cc:2258] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:49.465178 4969 ts_tablet_manager.cc:1905] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:49.465216 4968 ts_tablet_manager.cc:1905] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:49.466513 4968 ts_tablet_manager.cc:1918] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:49.466572 4968 log.cc:1196] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/553d4497fccf4a3ca92301fa4bd5229b
I20260620 09:16:49.466639 4969 ts_tablet_manager.cc:1918] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:49.466706 4969 log.cc:1196] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/553d4497fccf4a3ca92301fa4bd5229b
I20260620 09:16:49.466825 4968 ts_tablet_manager.cc:1939] T 553d4497fccf4a3ca92301fa4bd5229b P 1095055f896d416daeefb41c7e1af5d6: Deleting consensus metadata
I20260620 09:16:49.466950 4969 ts_tablet_manager.cc:1939] T 553d4497fccf4a3ca92301fa4bd5229b P 78a0c2a8e49b4e39ab9605172a63a9f4: Deleting consensus metadata
I20260620 09:16:49.467207 4460 catalog_manager.cc:4841] TS 1095055f896d416daeefb41c7e1af5d6 (127.0.98.3:34085): tablet 553d4497fccf4a3ca92301fa4bd5229b (table test-table [id=89b6ba29f43a4e1c830ff303739381c2]) successfully deleted
I20260620 09:16:49.467579 4458 catalog_manager.cc:4841] TS 78a0c2a8e49b4e39ab9605172a63a9f4 (127.0.98.2:33517): tablet 553d4497fccf4a3ca92301fa4bd5229b (table test-table [id=89b6ba29f43a4e1c830ff303739381c2]) successfully deleted
W20260620 09:16:49.470271 4461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46699 (error 108)
I20260620 09:16:49.470709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 4669
I20260620 09:16:49.470742 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.470620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46699 (local address 127.0.0.1:34712)
0620 09:16:49.470676 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.470716 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 09:16:49.471290 4461 catalog_manager.cc:4584] TS 2f06c99ebe114f45bcbcd79c560bf185 (127.0.98.1:46699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 553d4497fccf4a3ca92301fa4bd5229b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46699: connect: Connection refused (error 111)
I20260620 09:16:49.475337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 4802
I20260620 09:16:49.479070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 4443
2026-06-20T09:16:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1760 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T09:16:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:49Z Disabled control of system clock
I20260620 09:16:49.504001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46809
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 09:16:49.580341 4974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:49.580525 4974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:49.580544 4974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:49.581594 4974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:49.581624 4974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:49.581634 4974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:49.581645 4974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:49.581656 4974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:49.582881 4974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:49.583060 4974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:49.583276 4974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:49.587567 4982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:49.587594 4980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:49.587697 4979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:49.587757 4974 server_base.cc:1034] running on GCE node
I20260620 09:16:49.588047 4974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:49.588302 4974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:49.589444 4974 hybrid_clock.cc:648] HybridClock initialized: now 1781947009589424 us; error 32 us; skew 500 ppm
I20260620 09:16:49.590541 4974 webserver.cc:460] Webserver started at http://127.0.98.62:36559/ using document root <none> and password file <none>
I20260620 09:16:49.590750 4974 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:49.590795 4974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:49.590914 4974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:49.591939 4974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "37c269e83bb54fe280df3bbf0365f2f3"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "1e6f2a588533987380d2c8c561e8667c"
server_key_iv: "f9664bc61c15b98b9a8a1ba45740cb42"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.592247 4974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "37c269e83bb54fe280df3bbf0365f2f3"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "1e6f2a588533987380d2c8c561e8667c"
server_key_iv: "f9664bc61c15b98b9a8a1ba45740cb42"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.593375 4974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:49.593961 4988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.594141 4974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:49.594215 4974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "37c269e83bb54fe280df3bbf0365f2f3"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "1e6f2a588533987380d2c8c561e8667c"
server_key_iv: "f9664bc61c15b98b9a8a1ba45740cb42"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.594271 4974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:49.597412 4974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:49.597646 4974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:49.597774 4974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:49.601269 4974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46809
I20260620 09:16:49.601306 5040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46809 every 8 connection(s)
I20260620 09:16:49.601605 4974 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:49.602206 5041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:49.604796 5041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3: Bootstrap starting.
I20260620 09:16:49.605279 5041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:49.605500 5041 log.cc:824] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:49.606055 5041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3: No bootstrap required, opened a new log
I20260620 09:16:49.607467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 4974
I20260620 09:16:49.607560 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:49.607522 5041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c269e83bb54fe280df3bbf0365f2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46809 } }
I20260620 09:16:49.607719 5041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:49.607743 392 external_mini_cluster.cc:1428] Setting key 34450072af19b259aaf8e2ef4bc24c56
I20260620 09:16:49.607760 5041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37c269e83bb54fe280df3bbf0365f2f3, State: Initialized, Role: FOLLOWER
I20260620 09:16:49.607896 5041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c269e83bb54fe280df3bbf0365f2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46809 } }
I20260620 09:16:49.607972 5041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:49.607995 5041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:49.608045 5041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:49.608568 5041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c269e83bb54fe280df3bbf0365f2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46809 } }
I20260620 09:16:49.608656 5041 leader_election.cc:302] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37c269e83bb54fe280df3bbf0365f2f3; no voters:
I20260620 09:16:49.608801 5041 leader_election.cc:288] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:49.608950 5047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:49.609031 5041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:49.609076 5047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 37c269e83bb54fe280df3bbf0365f2f3, State: Running, Role: LEADER
I20260620 09:16:49.609160 5047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c269e83bb54fe280df3bbf0365f2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46809 } }
I20260620 09:16:49.609438 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37c269e83bb54fe280df3bbf0365f2f3. Latest consensus state: current_term: 1 leader_uuid: "37c269e83bb54fe280df3bbf0365f2f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c269e83bb54fe280df3bbf0365f2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46809 } } }
I20260620 09:16:49.609503 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:49.609766 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.608152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.0.98.62:46809)
0620 09:16:49.608344 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.608353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.608377 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:16:49.608467 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.608472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.608508 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.608618 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:49.609274 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:49.609279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:49.609279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:49.609283 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:49.609327 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:49.609342 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:49.609414 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.609497 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.609557 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260620 09:16:49.609988 5048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37c269e83bb54fe280df3bbf0365f2f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c269e83bb54fe280df3bbf0365f2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46809 } } }
I20260620 09:16:49.610064 5048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3 [sys.catalog]: This master's current role is: LEADER
W20260620 09:16:49.610828 5061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:16:49.610875 5061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:16:49.611028 5051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:49.611182 5051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:49.612460 5051 catalog_manager.cc:1347] Generated new cluster ID: e1d2b448d9cc48ca93dd621bdd29e99d
I20260620 09:16:49.612497 5051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:16:49.617419 5051 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:49.617552 5051 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:49.620936 5051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3: Generated new TSK 0
I20260620 09:16:49.621102 5051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:16:49.626825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46809
--builtin_ntp_servers=127.0.98.20:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 09:16:49.712026 5065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:49.712221 5065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:49.712251 5065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:49.713389 5065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:49.713425 5065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:49.713459 5065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:49.714790 5065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:49.715018 5065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:49.715281 5065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:49.720870 5071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:49.721091 5073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:49.721136 5065 server_base.cc:1034] running on GCE node
W20260620 09:16:49.720830 5070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:49.721398 5065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:49.721676 5065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:49.722852 5065 hybrid_clock.cc:648] HybridClock initialized: now 1781947009722840 us; error 25 us; skew 500 ppm
I20260620 09:16:49.724217 5065 webserver.cc:460] Webserver started at http://127.0.98.1:40279/ using document root <none> and password file <none>
I20260620 09:16:49.724473 5065 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:49.724537 5065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:49.724678 5065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:49.726001 5065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "cabfb66737944054824aeef3aefab9fa"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "330dcf8896b6fb10a825b93fed9a2d4a"
server_key_iv: "f7756449f7b7fbfee54c4ecc86b11107"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.726363 5065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cabfb66737944054824aeef3aefab9fa"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "330dcf8896b6fb10a825b93fed9a2d4a"
server_key_iv: "f7756449f7b7fbfee54c4ecc86b11107"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.727818 5065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:16:49.728549 5079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.728760 5065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.728840 5065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "cabfb66737944054824aeef3aefab9fa"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "330dcf8896b6fb10a825b93fed9a2d4a"
server_key_iv: "f7756449f7b7fbfee54c4ecc86b11107"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.728904 5065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:49.732007 5065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:49.732231 5065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:49.732338 5065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:49.732515 5065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:49.732781 5065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:49.732812 5065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.732836 5065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:49.732860 5065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.737874 5065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46245
I20260620 09:16:49.738209 5065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:16:49.738361 5192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46245 every 8 connection(s)
I20260620 09:16:49.741005 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.738890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48467 (local address 127.0.98.62:46809)
0620 09:16:49.738926 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.738928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.739180 (+ 252us) server_negotiation.cc:407] Connection header received
0620 09:16:49.739663 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.739664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.739671 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.739721 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:49.740782 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:49.740783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:49.740783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:49.740788 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:49.740804 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:49.740808 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:49.740856 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.740925 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.740939 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 09:16:49.741202 5195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.738892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46809 (local address 127.0.98.1:48467)
0620 09:16:49.739066 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.739099 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.739548 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.739788 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.739792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.739800 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:49.740644 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:16:49.740644 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:49.740671 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:16:49.740697 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:49.740858 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:49.740862 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:49.740865 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.740929 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.740984 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260620 09:16:49.741489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 5065
I20260620 09:16:49.741597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:16:49.741693 5193 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46809
I20260620 09:16:49.741761 392 external_mini_cluster.cc:1428] Setting key 1927e5a2bc9cd13a820f9315c7b00760
I20260620 09:16:49.741782 5193 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:49.741900 5193 heartbeater.cc:511] Master 127.0.98.62:46809 requested a full tablet report, sending...
I20260620 09:16:49.742338 5004 ts_manager.cc:194] Registered new tserver with Master: cabfb66737944054824aeef3aefab9fa (127.0.98.1:46245)
I20260620 09:16:49.742573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46809
--builtin_ntp_servers=127.0.98.20:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 09:16:49.819669 5196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:49.819864 5196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:49.819883 5196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:49.820995 5196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:49.821026 5196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:49.821049 5196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:49.822531 5196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:49.822724 5196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:49.822957 5196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:49.827350 5202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:49.827376 5201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:49.827373 5204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:49.827462 5196 server_base.cc:1034] running on GCE node
I20260620 09:16:49.827739 5196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:49.827943 5196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:49.829080 5196 hybrid_clock.cc:648] HybridClock initialized: now 1781947009829068 us; error 27 us; skew 500 ppm
I20260620 09:16:49.830128 5196 webserver.cc:460] Webserver started at http://127.0.98.2:36041/ using document root <none> and password file <none>
I20260620 09:16:49.830330 5196 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:49.830376 5196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:49.830497 5196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:49.831519 5196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "698343e22caf42c2a10047834201473c"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "c4b70fb85bf33c17a4c5c47acb5c8136"
server_key_iv: "291ed02660b809671218f8fe89a78bc5"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.831867 5196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "698343e22caf42c2a10047834201473c"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "c4b70fb85bf33c17a4c5c47acb5c8136"
server_key_iv: "291ed02660b809671218f8fe89a78bc5"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.833034 5196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:49.833711 5210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.833865 5196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:49.833949 5196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "698343e22caf42c2a10047834201473c"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "c4b70fb85bf33c17a4c5c47acb5c8136"
server_key_iv: "291ed02660b809671218f8fe89a78bc5"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.834007 5196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:49.837133 5196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:49.837358 5196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:49.837471 5196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:49.837641 5196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:49.837913 5196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:49.837944 5196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.837968 5196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:49.837992 5196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.843019 5196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46803
I20260620 09:16:49.843082 5323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46803 every 8 connection(s)
I20260620 09:16:49.843335 5196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:16:49.846031 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.843993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40623 (local address 127.0.98.62:46809)
0620 09:16:49.844040 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.844041 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.844251 (+ 210us) server_negotiation.cc:407] Connection header received
0620 09:16:49.844698 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.844699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.844706 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.844752 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:49.845804 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:49.845806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:49.845806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:49.845811 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:49.845827 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:49.845831 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:49.845880 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.845952 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.845966 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:16:49.846201 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.843986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46809 (local address 127.0.98.2:40623)
0620 09:16:49.844142 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.844173 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.844609 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.844831 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.844835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.844845 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:49.845669 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:16:49.845670 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:49.845695 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:16:49.845722 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:49.845884 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:49.845888 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:49.845892 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.845956 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.846013 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260620 09:16:49.846575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 5196
I20260620 09:16:49.846585 5324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46809
I20260620 09:16:49.846661 5324 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:49.846664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:16:49.846792 5324 heartbeater.cc:511] Master 127.0.98.62:46809 requested a full tablet report, sending...
I20260620 09:16:49.846822 392 external_mini_cluster.cc:1428] Setting key ee9d259271d9163d8eefee50e176ab1c
I20260620 09:16:49.847146 5005 ts_manager.cc:194] Registered new tserver with Master: 698343e22caf42c2a10047834201473c (127.0.98.2:46803)
I20260620 09:16:49.847791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46809
--builtin_ntp_servers=127.0.98.20:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 09:16:49.925065 5327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:49.925261 5327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:49.925282 5327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:49.926389 5327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:49.926424 5327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:49.926446 5327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:49.927735 5327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:49.927922 5327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:49.928154 5327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:49.933159 5335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:49.933166 5333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:49.933190 5332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:49.933535 5327 server_base.cc:1034] running on GCE node
I20260620 09:16:49.933743 5327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:49.933983 5327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:49.935148 5327 hybrid_clock.cc:648] HybridClock initialized: now 1781947009935131 us; error 29 us; skew 500 ppm
I20260620 09:16:49.936543 5327 webserver.cc:460] Webserver started at http://127.0.98.3:32811/ using document root <none> and password file <none>
I20260620 09:16:49.936758 5327 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:49.936805 5327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:49.936928 5327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:49.938009 5327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "b394fceb02784bca8e40197b00580d05"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "c5b589b51b12eff31118b92e483a6599"
server_key_iv: "ae1013ea61c6a431c65b3e8cd9008496"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.938351 5327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b394fceb02784bca8e40197b00580d05"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "c5b589b51b12eff31118b92e483a6599"
server_key_iv: "ae1013ea61c6a431c65b3e8cd9008496"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.939604 5327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:16:49.940341 5341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.940495 5327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:49.940585 5327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "b394fceb02784bca8e40197b00580d05"
format_stamp: "Formatted at 2026-06-20 09:16:49 on dist-test-slave-rq8c"
server_key: "c5b589b51b12eff31118b92e483a6599"
server_key_iv: "ae1013ea61c6a431c65b3e8cd9008496"
server_key_version: "encryptionkey@0"
I20260620 09:16:49.940644 5327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:49.943876 5327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:49.944123 5327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:49.944237 5327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:49.944425 5327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:49.944707 5327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:49.944738 5327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.944763 5327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:49.944787 5327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:49.949939 5327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45321
I20260620 09:16:49.949975 5454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45321 every 8 connection(s)
I20260620 09:16:49.950265 5327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:16:49.951880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 5327
I20260620 09:16:49.951988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:16:49.952164 392 external_mini_cluster.cc:1428] Setting key ef9fa39f3138c5d93b32930462104fb3
I20260620 09:16:49.954103 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.950922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52837 (local address 127.0.98.62:46809)
0620 09:16:49.950980 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.950981 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.951183 (+ 202us) server_negotiation.cc:407] Connection header received
0620 09:16:49.951648 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.951649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.951655 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.951722 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:49.953912 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:49.953913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:49.953913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:49.953917 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:49.953931 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:49.953935 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:49.953980 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.954036 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.954052 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:16:49.954268 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.950895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46809 (local address 127.0.98.3:52837)
0620 09:16:49.951063 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.951101 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.951553 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.951797 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.951801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.951811 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:49.953781 (+ 1970us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:16:49.953781 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:49.953811 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:16:49.953839 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:49.953980 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:49.953982 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:49.953985 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.954031 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.954083 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":81,"threads_started":1}
I20260620 09:16:49.954675 5455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46809
I20260620 09:16:49.954756 5455 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:49.954865 5455 heartbeater.cc:511] Master 127.0.98.62:46809 requested a full tablet report, sending...
I20260620 09:16:49.955180 5004 ts_manager.cc:194] Registered new tserver with Master: b394fceb02784bca8e40197b00580d05 (127.0.98.3:45321)
I20260620 09:16:49.963867 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:16:49.966042 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.965188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.0.98.62:46809)
0620 09:16:49.965233 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.965235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.965271 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:16:49.965335 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.965336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.965340 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.965376 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:49.965864 (+ 488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:49.965865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:49.965865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:49.965868 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:49.965884 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:49.965887 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:49.965925 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.965960 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.965992 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:16:49.967396 5004 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:49.968933 5004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:49.972790 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.971290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46245 (local address 127.0.0.1:52940)
0620 09:16:49.971447 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.971456 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.971488 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.971881 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.971881 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.971883 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:49.972336 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:16:49.972337 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:49.972353 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:16:49.972377 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:49.972625 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:49.972632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:49.972636 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.972704 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.972717 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":57,"threads_started":1}
I20260620 09:16:49.972925 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.971538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.0.98.2:46803)
0620 09:16:49.971714 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.971718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.971770 (+ 52us) server_negotiation.cc:407] Connection header received
0620 09:16:49.971825 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.971828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.971836 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.971895 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:49.972603 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:49.972610 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:49.972610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:49.972614 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:49.972644 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:49.972669 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:49.972717 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.972826 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.972841 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260620 09:16:49.972983 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.971399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.0.98.1:46245)
0620 09:16:49.971597 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.971601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.971610 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:16:49.971764 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.971770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.971780 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.971875 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:49.972454 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:49.972464 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:49.972464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:49.972468 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:49.972506 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:49.972542 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:49.972627 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.972891 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.972909 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
I20260620 09:16:49.973140 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.971088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45321 (local address 127.0.0.1:53196)
0620 09:16:49.971246 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.971289 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.971348 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.971829 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.971835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.971841 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:49.972672 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:16:49.972673 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:49.972679 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:16:49.972689 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:49.973039 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:49.973039 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:49.973040 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.973068 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.973080 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":59,"threads_started":1}
I20260620 09:16:49.973243 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.971425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46803 (local address 127.0.0.1:37864)
0620 09:16:49.971720 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:49.971729 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:49.971774 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:49.972183 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:49.972184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:49.972186 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:49.972505 (+ 319us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:16:49.972506 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:49.972512 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:16:49.972522 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:49.972739 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:49.972740 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:49.972740 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 09:16:49.972778 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 09:16:49.972790 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":54,"threads_started":1}
I20260620 09:16:49.973449 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:49.971204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.0.98.3:45321)
0620 09:16:49.971568 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:49.971573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:49.971583 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:16:49.971609 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:49.971611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:49.971642 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:49.972664 (+ 1022us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:49.972916 (+ 252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:49.972920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:49.972920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:49.972923 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:49.972949 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:49.972962 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:49.973335 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:49.973358 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 09:16:49.973373 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":105,"threads_started":1}
I20260620 09:16:49.973829 5258 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbc0bede445479f9df6d4b70293b29d (DEFAULT_TABLE table=test-table [id=51c7e6a302fb40e2bbd15ea7f1644071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:49.973790 5127 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbc0bede445479f9df6d4b70293b29d (DEFAULT_TABLE table=test-table [id=51c7e6a302fb40e2bbd15ea7f1644071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:49.974010 5389 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbc0bede445479f9df6d4b70293b29d (DEFAULT_TABLE table=test-table [id=51c7e6a302fb40e2bbd15ea7f1644071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:49.974136 5258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbc0bede445479f9df6d4b70293b29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:49.974139 5127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbc0bede445479f9df6d4b70293b29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:49.974313 5389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbc0bede445479f9df6d4b70293b29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:49.976663 5469 tablet_bootstrap.cc:492] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c: Bootstrap starting.
I20260620 09:16:49.977311 5469 tablet_bootstrap.cc:654] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:49.977458 5470 tablet_bootstrap.cc:492] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05: Bootstrap starting.
I20260620 09:16:49.977634 5469 log.cc:824] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:49.977878 5471 tablet_bootstrap.cc:492] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa: Bootstrap starting.
I20260620 09:16:49.978099 5470 tablet_bootstrap.cc:654] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:49.978358 5469 tablet_bootstrap.cc:492] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c: No bootstrap required, opened a new log
I20260620 09:16:49.978387 5470 log.cc:824] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:49.978425 5469 ts_tablet_manager.cc:1397] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:49.978572 5471 tablet_bootstrap.cc:654] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:49.978842 5471 log.cc:824] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:49.979089 5470 tablet_bootstrap.cc:492] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05: No bootstrap required, opened a new log
I20260620 09:16:49.979143 5470 ts_tablet_manager.cc:1397] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:49.979489 5471 tablet_bootstrap.cc:492] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa: No bootstrap required, opened a new log
I20260620 09:16:49.979562 5471 ts_tablet_manager.cc:1397] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:49.980530 5469 raft_consensus.cc:348] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b394fceb02784bca8e40197b00580d05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "cabfb66737944054824aeef3aefab9fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46245 } } peers { permanent_uuid: "698343e22caf42c2a10047834201473c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } }
I20260620 09:16:49.980664 5469 raft_consensus.cc:374] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:49.980695 5469 raft_consensus.cc:729] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698343e22caf42c2a10047834201473c, State: Initialized, Role: FOLLOWER
I20260620 09:16:49.980826 5469 consensus_queue.cc:260] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b394fceb02784bca8e40197b00580d05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "cabfb66737944054824aeef3aefab9fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46245 } } peers { permanent_uuid: "698343e22caf42c2a10047834201473c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } }
I20260620 09:16:49.981101 5469 ts_tablet_manager.cc:1428] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:16:49.981222 5324 heartbeater.cc:503] Master 127.0.98.62:46809 was elected leader, sending a full tablet report...
I20260620 09:16:49.981974 5471 raft_consensus.cc:348] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b394fceb02784bca8e40197b00580d05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "cabfb66737944054824aeef3aefab9fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46245 } } peers { permanent_uuid: "698343e22caf42c2a10047834201473c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } }
I20260620 09:16:49.981911 5470 raft_consensus.cc:348] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b394fceb02784bca8e40197b00580d05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "cabfb66737944054824aeef3aefab9fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46245 } } peers { permanent_uuid: "698343e22caf42c2a10047834201473c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } }
I20260620 09:16:49.982120 5470 raft_consensus.cc:374] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:49.982120 5471 raft_consensus.cc:374] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:49.982160 5470 raft_consensus.cc:729] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b394fceb02784bca8e40197b00580d05, State: Initialized, Role: FOLLOWER
I20260620 09:16:49.982160 5471 raft_consensus.cc:729] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabfb66737944054824aeef3aefab9fa, State: Initialized, Role: FOLLOWER
I20260620 09:16:49.982270 5470 consensus_queue.cc:260] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b394fceb02784bca8e40197b00580d05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "cabfb66737944054824aeef3aefab9fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46245 } } peers { permanent_uuid: "698343e22caf42c2a10047834201473c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } }
I20260620 09:16:49.982270 5471 consensus_queue.cc:260] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b394fceb02784bca8e40197b00580d05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "cabfb66737944054824aeef3aefab9fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46245 } } peers { permanent_uuid: "698343e22caf42c2a10047834201473c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } }
I20260620 09:16:49.982503 5471 ts_tablet_manager.cc:1428] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 09:16:49.982503 5470 ts_tablet_manager.cc:1428] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:16:49.982539 5455 heartbeater.cc:503] Master 127.0.98.62:46809 was elected leader, sending a full tablet report...
I20260620 09:16:49.982765 5193 heartbeater.cc:503] Master 127.0.98.62:46809 was elected leader, sending a full tablet report...
W20260620 09:16:49.988718 5194 tablet.cc:2369] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:16:50.093909 5325 tablet.cc:2369] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:16:50.127826 5477 raft_consensus.cc:482] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:50.127955 5477 raft_consensus.cc:504] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b394fceb02784bca8e40197b00580d05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "cabfb66737944054824aeef3aefab9fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46245 } } peers { permanent_uuid: "698343e22caf42c2a10047834201473c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } }
I20260620 09:16:50.128254 5477 leader_election.cc:288] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cabfb66737944054824aeef3aefab9fa (127.0.98.1:46245), 698343e22caf42c2a10047834201473c (127.0.98.2:46803)
I20260620 09:16:50.129473 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.128442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46245 (local address 127.0.98.3:50181)
0620 09:16:50.128500 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:50.128513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:50.128571 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:50.128717 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:50.128718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:50.128721 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:50.129174 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:16:50.129175 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:50.129181 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:16:50.129192 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:50.129339 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:50.129340 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:50.129341 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 09:16:50.129388 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:16:50.129402 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 09:16:50.129473 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.128526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50181 (local address 127.0.98.1:46245)
0620 09:16:50.128568 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:50.128569 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:50.128579 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:16:50.128635 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:50.128636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:50.128640 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:50.128696 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:50.129264 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:50.129266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:50.129266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:50.129271 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:50.129288 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:50.129292 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:50.129338 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:50.129391 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 09:16:50.129403 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:16:50.129674 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.128630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45203 (local address 127.0.98.2:46803)
0620 09:16:50.128683 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:50.128685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:50.128729 (+ 44us) server_negotiation.cc:407] Connection header received
0620 09:16:50.128776 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:50.128777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:50.128782 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:50.128842 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:50.129419 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:50.129420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:50.129420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:50.129425 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:50.129441 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:50.129446 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:50.129491 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:50.129595 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 09:16:50.129608 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:16:50.129834 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.128539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46803 (local address 127.0.98.3:45203)
0620 09:16:50.128684 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:50.128692 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:50.128732 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:50.128845 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:50.128846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:50.128848 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:50.129298 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:16:50.129299 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:50.129306 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:16:50.129317 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:50.129533 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:50.129534 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:50.129535 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 09:16:50.129754 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 09:16:50.129768 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260620 09:16:50.129820 5147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbc0bede445479f9df6d4b70293b29d" candidate_uuid: "b394fceb02784bca8e40197b00580d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cabfb66737944054824aeef3aefab9fa" is_pre_election: true
I20260620 09:16:50.129963 5147 raft_consensus.cc:2454] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b394fceb02784bca8e40197b00580d05 in term 0.
I20260620 09:16:50.130134 5278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbc0bede445479f9df6d4b70293b29d" candidate_uuid: "b394fceb02784bca8e40197b00580d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698343e22caf42c2a10047834201473c" is_pre_election: true
I20260620 09:16:50.130172 5342 leader_election.cc:302] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b394fceb02784bca8e40197b00580d05, cabfb66737944054824aeef3aefab9fa; no voters:
I20260620 09:16:50.130265 5278 raft_consensus.cc:2454] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b394fceb02784bca8e40197b00580d05 in term 0.
I20260620 09:16:50.130287 5477 raft_consensus.cc:2781] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:50.130350 5477 raft_consensus.cc:482] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:50.130379 5477 raft_consensus.cc:3037] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:50.131050 5477 raft_consensus.cc:504] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b394fceb02784bca8e40197b00580d05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "cabfb66737944054824aeef3aefab9fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46245 } } peers { permanent_uuid: "698343e22caf42c2a10047834201473c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } }
I20260620 09:16:50.131157 5477 leader_election.cc:288] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [CANDIDATE]: Term 1 election: Requested vote from peers cabfb66737944054824aeef3aefab9fa (127.0.98.1:46245), 698343e22caf42c2a10047834201473c (127.0.98.2:46803)
I20260620 09:16:50.131323 5147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbc0bede445479f9df6d4b70293b29d" candidate_uuid: "b394fceb02784bca8e40197b00580d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cabfb66737944054824aeef3aefab9fa"
I20260620 09:16:50.131354 5278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbc0bede445479f9df6d4b70293b29d" candidate_uuid: "b394fceb02784bca8e40197b00580d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "698343e22caf42c2a10047834201473c"
I20260620 09:16:50.131392 5147 raft_consensus.cc:3037] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:50.131423 5278 raft_consensus.cc:3037] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:50.132114 5147 raft_consensus.cc:2454] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b394fceb02784bca8e40197b00580d05 in term 1.
I20260620 09:16:50.132205 5278 raft_consensus.cc:2454] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b394fceb02784bca8e40197b00580d05 in term 1.
I20260620 09:16:50.132268 5342 leader_election.cc:302] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b394fceb02784bca8e40197b00580d05, cabfb66737944054824aeef3aefab9fa; no voters:
I20260620 09:16:50.132373 5477 raft_consensus.cc:2781] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:50.132501 5477 raft_consensus.cc:686] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 1 LEADER]: Becoming Leader. State: Replica: b394fceb02784bca8e40197b00580d05, State: Running, Role: LEADER
I20260620 09:16:50.132575 5477 consensus_queue.cc:237] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b394fceb02784bca8e40197b00580d05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "cabfb66737944054824aeef3aefab9fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46245 } } peers { permanent_uuid: "698343e22caf42c2a10047834201473c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } }
I20260620 09:16:50.133356 5005 catalog_manager.cc:5495] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 reported cstate change: term changed from 0 to 1, leader changed from <none> to b394fceb02784bca8e40197b00580d05 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b394fceb02784bca8e40197b00580d05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b394fceb02784bca8e40197b00580d05" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabfb66737944054824aeef3aefab9fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "698343e22caf42c2a10047834201473c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 } health_report { overall_health: UNKNOWN } } }
I20260620 09:16:50.154733 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.153918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.0.98.3:45321)
0620 09:16:50.153961 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:50.153963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:50.153972 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:16:50.154007 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:50.154007 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:50.154020 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:50.154069 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:50.154555 (+ 486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:50.154556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:50.154556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:50.154560 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:50.154574 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:50.154577 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:50.154618 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:50.154651 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 09:16:50.154661 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:16:50.156503 5278 raft_consensus.cc:1261] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c [term 1 FOLLOWER]: Refusing update from remote peer b394fceb02784bca8e40197b00580d05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:50.156592 5147 raft_consensus.cc:1261] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa [term 1 FOLLOWER]: Refusing update from remote peer b394fceb02784bca8e40197b00580d05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:50.156847 5477 consensus_queue.cc:1035] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "698343e22caf42c2a10047834201473c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:50.157014 5479 consensus_queue.cc:1035] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabfb66737944054824aeef3aefab9fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:50.159389 5483 mvcc.cc:205] Tried to move back new op lower bound from 7298854953598607360 to 7298854953503707136. Current Snapshot: MvccSnapshot[applied={T|T < 7298854953598607360}]
I20260620 09:16:50.159961 5484 mvcc.cc:205] Tried to move back new op lower bound from 7298854953598607360 to 7298854953503707136. Current Snapshot: MvccSnapshot[applied={T|T < 7298854953598607360}]
I20260620 09:16:50.161617 5005 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:50.161713 5005 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:50.162392 5005 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37c269e83bb54fe280df3bbf0365f2f3: Sending DeleteTablet for 3 replicas of tablet 7dbc0bede445479f9df6d4b70293b29d
I20260620 09:16:50.162664 5389 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbc0bede445479f9df6d4b70293b29d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:50 UTC) from {username='slave'} at 127.0.0.1:53196
I20260620 09:16:50.162712 5127 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbc0bede445479f9df6d4b70293b29d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:50 UTC) from {username='slave'} at 127.0.0.1:52940
I20260620 09:16:50.162760 5258 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbc0bede445479f9df6d4b70293b29d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:50 UTC) from {username='slave'} at 127.0.0.1:37864
I20260620 09:16:50.162917 5492 tablet_replica.cc:330] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa: stopping tablet replica
I20260620 09:16:50.162938 5493 tablet_replica.cc:330] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c: stopping tablet replica
I20260620 09:16:50.163004 5492 raft_consensus.cc:2229] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:50.163115 5492 raft_consensus.cc:2258] T 7dbc0bede445479f9df6d4b70293b29d P cabfb66737944054824aeef3aefab9fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:50.163112 5491 tablet_replica.cc:330] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05: stopping tablet replica
I20260620 09:16:50.163188 5491 raft_consensus.cc:2229] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:50.163182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 5065
I20260620 09:16:50.163262 5491 raft_consensus.cc:2258] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:50.164151 5491 ts_tablet_manager.cc:1905] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:50.165436 5491 ts_tablet_manager.cc:1918] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:50.165478 5491 log.cc:1196] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/7dbc0bede445479f9df6d4b70293b29d
I20260620 09:16:50.165664 5491 ts_tablet_manager.cc:1939] T 7dbc0bede445479f9df6d4b70293b29d P b394fceb02784bca8e40197b00580d05: Deleting consensus metadata
I20260620 09:16:50.165951 4989 catalog_manager.cc:4841] TS b394fceb02784bca8e40197b00580d05 (127.0.98.3:45321): tablet 7dbc0bede445479f9df6d4b70293b29d (table test-table [id=51c7e6a302fb40e2bbd15ea7f1644071]) successfully deleted
I20260620 09:16:50.168190 5493 raft_consensus.cc:2229] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:50.168318 5493 raft_consensus.cc:2258] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:50.168777 5493 ts_tablet_manager.cc:1905] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:50.170085 5493 ts_tablet_manager.cc:1918] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:50.170147 5493 log.cc:1196] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/7dbc0bede445479f9df6d4b70293b29d
I20260620 09:16:50.170384 5493 ts_tablet_manager.cc:1939] T 7dbc0bede445479f9df6d4b70293b29d P 698343e22caf42c2a10047834201473c: Deleting consensus metadata
W20260620 09:16:50.170475 4989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46245 (error 108)
I20260620 09:16:50.171025 4989 catalog_manager.cc:4841] TS 698343e22caf42c2a10047834201473c (127.0.98.2:46803): tablet 7dbc0bede445479f9df6d4b70293b29d (table test-table [id=51c7e6a302fb40e2bbd15ea7f1644071]) successfully deleted
I20260620 09:16:50.171096 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.170978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46245 (local address 127.0.0.1:52946)
0620 09:16:50.171027 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:50.171064 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 09:16:50.171181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 5196
W20260620 09:16:50.171201 4989 catalog_manager.cc:4584] TS cabfb66737944054824aeef3aefab9fa (127.0.98.1:46245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7dbc0bede445479f9df6d4b70293b29d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46245: connect: Connection refused (error 111)
I20260620 09:16:50.178074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 5327
I20260620 09:16:50.182375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 4974
2026-06-20T09:16:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (702 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 09:16:50.200407 392 mini_oidc.cc:152] Starting JWKS server
I20260620 09:16:50.201408 392 webserver.cc:460] Webserver started at https://127.0.0.1:33893/ using document root <none> and password file <none>
I20260620 09:16:50.201495 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 09:16:50.201647 392 webserver.cc:460] Webserver started at http://127.0.0.1:42841/ using document root <none> and password file <none>
2026-06-20T09:16:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:50Z Disabled control of system clock
I20260620 09:16:50.208400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42791
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42791
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33893/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/testchainca.cert with env {}
W20260620 09:16:50.285493 5500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:50.285679 5500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:50.285699 5500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:50.286784 5500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:50.286815 5500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:50.286827 5500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:50.286839 5500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:50.286849 5500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:16:50.286859 5500 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:16:50.286873 5500 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33893/jwks.json
I20260620 09:16:50.288331 5500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33893/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:50.288506 5500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:50.288648 5500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:50.292963 5506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:50.292985 5508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:50.292989 5505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:50.293172 5500 server_base.cc:1034] running on GCE node
I20260620 09:16:50.293434 5500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:50.293686 5500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:50.294822 5500 hybrid_clock.cc:648] HybridClock initialized: now 1781947010294809 us; error 28 us; skew 500 ppm
I20260620 09:16:50.295897 5500 webserver.cc:460] Webserver started at http://127.0.98.62:41369/ using document root <none> and password file <none>
I20260620 09:16:50.296087 5500 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:50.296131 5500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:50.296244 5500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:50.297199 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "46ac5f1cd6c140e994e22cf113c2c38c"
format_stamp: "Formatted at 2026-06-20 09:16:50 on dist-test-slave-rq8c"
server_key: "73a92c3a912a357d2eb27da5b786db8e"
server_key_iv: "b1a3a1531784dbcc82ce9bb74553b99e"
server_key_version: "encryptionkey@0"
I20260620 09:16:50.297513 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "46ac5f1cd6c140e994e22cf113c2c38c"
format_stamp: "Formatted at 2026-06-20 09:16:50 on dist-test-slave-rq8c"
server_key: "73a92c3a912a357d2eb27da5b786db8e"
server_key_iv: "b1a3a1531784dbcc82ce9bb74553b99e"
server_key_version: "encryptionkey@0"
I20260620 09:16:50.298645 5500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:16:50.299239 5514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:50.299393 5500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:50.299468 5500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "46ac5f1cd6c140e994e22cf113c2c38c"
format_stamp: "Formatted at 2026-06-20 09:16:50 on dist-test-slave-rq8c"
server_key: "73a92c3a912a357d2eb27da5b786db8e"
server_key_iv: "b1a3a1531784dbcc82ce9bb74553b99e"
server_key_version: "encryptionkey@0"
I20260620 09:16:50.299536 5500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:50.325060 5500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:50.325341 5500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:50.325460 5500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:50.328866 5500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42791
I20260620 09:16:50.328898 5566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42791 every 8 connection(s)
I20260620 09:16:50.329193 5500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:50.330106 5567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:50.332119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 5500
I20260620 09:16:50.332195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:50.332257 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c: Bootstrap starting.
I20260620 09:16:50.332348 392 external_mini_cluster.cc:1428] Setting key 59830610bb001f570498578f9dacf1a4
I20260620 09:16:50.332831 5567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:50.333094 5567 log.cc:824] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:50.333670 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c: No bootstrap required, opened a new log
I20260620 09:16:50.335316 5567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ac5f1cd6c140e994e22cf113c2c38c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42791 } }
I20260620 09:16:50.335439 5567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:50.335469 5567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46ac5f1cd6c140e994e22cf113c2c38c, State: Initialized, Role: FOLLOWER
I20260620 09:16:50.335589 5567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ac5f1cd6c140e994e22cf113c2c38c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42791 } }
I20260620 09:16:50.335686 5567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:50.335718 5567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:50.335763 5567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:50.336249 5567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ac5f1cd6c140e994e22cf113c2c38c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42791 } }
I20260620 09:16:50.336336 5567 leader_election.cc:302] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46ac5f1cd6c140e994e22cf113c2c38c; no voters:
I20260620 09:16:50.336477 5567 leader_election.cc:288] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:50.336513 5572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:50.336611 5572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [term 1 LEADER]: Becoming Leader. State: Replica: 46ac5f1cd6c140e994e22cf113c2c38c, State: Running, Role: LEADER
I20260620 09:16:50.336694 5572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ac5f1cd6c140e994e22cf113c2c38c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42791 } }
I20260620 09:16:50.336735 5567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:50.337020 5572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46ac5f1cd6c140e994e22cf113c2c38c. Latest consensus state: current_term: 1 leader_uuid: "46ac5f1cd6c140e994e22cf113c2c38c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ac5f1cd6c140e994e22cf113c2c38c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42791 } } }
I20260620 09:16:50.337086 5572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:50.337047 5574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46ac5f1cd6c140e994e22cf113c2c38c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ac5f1cd6c140e994e22cf113c2c38c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42791 } } }
I20260620 09:16:50.337142 5574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:50.337431 5578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:50.337527 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.332760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.0.98.62:42791)
0620 09:16:50.332963 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:50.332970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:50.332993 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:16:50.333389 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:50.333396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:50.333451 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:50.333579 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:50.333764 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.334609 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.335345 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.335857 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.335966 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:50.336593 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:50.336597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:50.336597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:50.336786 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:50.336825 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:50.336839 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:50.336891 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:50.337228 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 09:16:50.337309 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260620 09:16:50.337890 5578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:50.338598 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:16:50.339529 5578 catalog_manager.cc:1347] Generated new cluster ID: 82061348825d4f309eba3802a9a65210
I20260620 09:16:50.339581 5578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:16:50.386945 5578 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:50.387666 5578 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:50.391752 5578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46ac5f1cd6c140e994e22cf113c2c38c: Generated new TSK 0
I20260620 09:16:50.391916 5578 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 09:16:50.416700 5597 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 09:16:50.416780 5597 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 09:16:50.417053 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.401969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.0.98.62:42791)
0620 09:16:50.402017 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:50.402019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:50.402127 (+ 108us) server_negotiation.cc:407] Connection header received
0620 09:16:50.402192 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:50.402193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:50.402204 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:50.402280 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:16:50.402371 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.403114 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.403861 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.404178 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.404240 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:50.404380 (+ 140us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:16:50.416824 (+ 12444us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 09:16:50.416971 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 09:16:50.417111 5570 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 09:16:50.417448 5596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:16:50.401901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42791 (local address 127.0.0.1:38796)
0620 09:16:50.402068 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:50.402076 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:50.402130 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:50.402281 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:50.402282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:50.402283 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:16:50.402333 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:50.402335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.403182 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.403182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:50.403797 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:50.403799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.404240 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.404242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:50.404280 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:50.404296 (+ 16us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:16:50.416977 (+ 12681us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 09:16:50.417080 (+ 103us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42791: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:42791: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 09:16:50.417935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 5500
2026-06-20T09:16:50Z chronyd exiting
I20260620 09:16:50.453100 392 test_util.cc:175] -----------------------------------------------
I20260620 09:16:50.453174 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946986834672-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (253 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 09:16:50.453882 392 mini_oidc.cc:152] Starting JWKS server
I20260620 09:16:50.454535 392 webserver.cc:460] Webserver started at https://127.0.0.1:38641/ using document root <none> and password file <none>
I20260620 09:16:50.454579 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 09:16:50.454711 392 webserver.cc:460] Webserver started at http://127.0.0.1:42369/ using document root <none> and password file <none>
2026-06-20T09:16:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:50Z Disabled control of system clock
I20260620 09:16:50.461174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36491
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36491
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38641/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/testchainca.cert with env {}
W20260620 09:16:50.537081 5603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:50.537271 5603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:50.537298 5603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:50.538412 5603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:50.538450 5603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:50.538466 5603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:50.538482 5603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:50.538507 5603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:16:50.538530 5603 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:16:50.538547 5603 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38641/jwks.json
I20260620 09:16:50.540009 5603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38641/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:50.540232 5603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:50.540406 5603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:50.544854 5609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:50.544945 5611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:50.544879 5608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:50.545048 5603 server_base.cc:1034] running on GCE node
I20260620 09:16:50.545256 5603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:50.545491 5603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:50.546627 5603 hybrid_clock.cc:648] HybridClock initialized: now 1781947010546618 us; error 26 us; skew 500 ppm
I20260620 09:16:50.547717 5603 webserver.cc:460] Webserver started at http://127.0.98.62:36197/ using document root <none> and password file <none>
I20260620 09:16:50.547917 5603 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:50.547959 5603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:50.548094 5603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:50.549064 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "8cf72b2db3404700a576576e1537fb7e"
format_stamp: "Formatted at 2026-06-20 09:16:50 on dist-test-slave-rq8c"
server_key: "31646b7e032cf664b74236b464a9cd35"
server_key_iv: "77d9ffb86e786b3458fcaaf9f3933dbb"
server_key_version: "encryptionkey@0"
I20260620 09:16:50.549374 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "8cf72b2db3404700a576576e1537fb7e"
format_stamp: "Formatted at 2026-06-20 09:16:50 on dist-test-slave-rq8c"
server_key: "31646b7e032cf664b74236b464a9cd35"
server_key_iv: "77d9ffb86e786b3458fcaaf9f3933dbb"
server_key_version: "encryptionkey@0"
I20260620 09:16:50.550495 5603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:16:50.551092 5617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:50.551235 5603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:50.551316 5603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "8cf72b2db3404700a576576e1537fb7e"
format_stamp: "Formatted at 2026-06-20 09:16:50 on dist-test-slave-rq8c"
server_key: "31646b7e032cf664b74236b464a9cd35"
server_key_iv: "77d9ffb86e786b3458fcaaf9f3933dbb"
server_key_version: "encryptionkey@0"
I20260620 09:16:50.551368 5603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:50.569011 5603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:50.569308 5603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:50.569429 5603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:50.573200 5603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36491
I20260620 09:16:50.573233 5669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36491 every 8 connection(s)
I20260620 09:16:50.573561 5603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:50.574452 5670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:50.574580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 5603
I20260620 09:16:50.574676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:50.574888 392 external_mini_cluster.cc:1428] Setting key 1b4e41542906dc4e9d681c9e4e83e71f
I20260620 09:16:50.577230 5670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e: Bootstrap starting.
I20260620 09:16:50.577775 5670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:50.578015 5670 log.cc:824] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:50.578583 5670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e: No bootstrap required, opened a new log
I20260620 09:16:50.579792 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.575340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.0.98.62:36491)
0620 09:16:50.575526 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:50.575533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:50.575557 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:16:50.576016 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:50.576023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:50.576084 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:50.576210 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:50.576408 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.577233 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.578032 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.578495 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.578602 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:50.579220 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:50.579224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:50.579224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:50.579373 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:50.579403 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:50.579413 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:50.579462 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:50.579538 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 09:16:50.579634 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260620 09:16:50.580591 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:16:50.580495 5670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf72b2db3404700a576576e1537fb7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36491 } }
I20260620 09:16:50.580636 5670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:50.580677 5670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf72b2db3404700a576576e1537fb7e, State: Initialized, Role: FOLLOWER
I20260620 09:16:50.580786 5670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf72b2db3404700a576576e1537fb7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36491 } }
I20260620 09:16:50.580852 5670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:50.580886 5670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:50.580929 5670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:50.581453 5670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf72b2db3404700a576576e1537fb7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36491 } }
I20260620 09:16:50.581548 5670 leader_election.cc:302] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cf72b2db3404700a576576e1537fb7e; no voters:
I20260620 09:16:50.581684 5670 leader_election.cc:288] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:50.581723 5675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:50.581848 5675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [term 1 LEADER]: Becoming Leader. State: Replica: 8cf72b2db3404700a576576e1537fb7e, State: Running, Role: LEADER
I20260620 09:16:50.581885 5670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:50.581938 5675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf72b2db3404700a576576e1537fb7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36491 } }
I20260620 09:16:50.582257 5677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cf72b2db3404700a576576e1537fb7e. Latest consensus state: current_term: 1 leader_uuid: "8cf72b2db3404700a576576e1537fb7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf72b2db3404700a576576e1537fb7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36491 } } }
I20260620 09:16:50.582341 5677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:50.582250 5676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cf72b2db3404700a576576e1537fb7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf72b2db3404700a576576e1537fb7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36491 } } }
I20260620 09:16:50.582489 5676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:50.582787 5682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:50.583223 5682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:50.584700 5682 catalog_manager.cc:1347] Generated new cluster ID: 6af7a40b8306495ab59d2327c0d41864
I20260620 09:16:50.584751 5682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:16:50.595698 5682 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:50.596167 5682 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:50.599037 5682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cf72b2db3404700a576576e1537fb7e: Generated new TSK 0
I20260620 09:16:50.599176 5682 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 09:16:50.623488 5700 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 09:16:50.623564 5700 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 09:16:50.623869 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.609909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.0.98.62:36491)
0620 09:16:50.609962 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:50.609964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:50.610067 (+ 103us) server_negotiation.cc:407] Connection header received
0620 09:16:50.610142 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:50.610143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:50.610153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:50.610208 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:16:50.610338 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.610811 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.611948 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.612139 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.612195 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:50.612350 (+ 155us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:16:50.623686 (+ 11336us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 09:16:50.623822 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32930: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 09:16:50.623919 5673 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32930: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 09:16:50.624027 5699 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:16:50.609855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36491 (local address 127.0.0.1:32930)
0620 09:16:50.610000 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:50.610012 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:50.610078 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:50.610225 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:50.610227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:50.610228 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:16:50.610285 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:50.610287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.610879 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.610880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:50.611856 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:50.611859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.612202 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.612206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:50.612264 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:50.612270 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:16:50.623802 (+ 11532us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 09:16:50.623914 (+ 112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36491: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260620 09:16:50.624305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 5603
2026-06-20T09:16:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (206 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 09:16:50.660552 392 mini_oidc.cc:152] Starting JWKS server
I20260620 09:16:50.661196 392 webserver.cc:460] Webserver started at https://127.0.0.1:35843/ using document root <none> and password file <none>
I20260620 09:16:50.661240 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 09:16:50.661366 392 webserver.cc:460] Webserver started at http://127.0.0.1:36767/ using document root <none> and password file <none>
2026-06-20T09:16:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:50Z Disabled control of system clock
I20260620 09:16:50.667793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33761
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33761
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35843/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 09:16:50.743120 5706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:50.743306 5706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:50.743335 5706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:50.744473 5706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:50.744511 5706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:50.744529 5706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:50.744545 5706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:50.744560 5706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:16:50.744581 5706 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:16:50.744601 5706 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35843/jwks.json
I20260620 09:16:50.745891 5706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35843/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:50.746114 5706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:50.746364 5706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:50.750703 5711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:50.750671 5712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:50.750870 5706 server_base.cc:1034] running on GCE node
W20260620 09:16:50.750672 5714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:50.751125 5706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:50.751379 5706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:50.752507 5706 hybrid_clock.cc:648] HybridClock initialized: now 1781947010752494 us; error 26 us; skew 500 ppm
I20260620 09:16:50.753568 5706 webserver.cc:460] Webserver started at http://127.0.98.62:42561/ using document root <none> and password file <none>
I20260620 09:16:50.753772 5706 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:50.753818 5706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:50.753935 5706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:50.754900 5706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "44773593f63a4f68abd4b8442a50ab6d"
format_stamp: "Formatted at 2026-06-20 09:16:50 on dist-test-slave-rq8c"
server_key: "3b23c6e14dd827f849c955181d59c3a7"
server_key_iv: "c5ef028de48e419bf6e152f98f5469d7"
server_key_version: "encryptionkey@0"
I20260620 09:16:50.755219 5706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "44773593f63a4f68abd4b8442a50ab6d"
format_stamp: "Formatted at 2026-06-20 09:16:50 on dist-test-slave-rq8c"
server_key: "3b23c6e14dd827f849c955181d59c3a7"
server_key_iv: "c5ef028de48e419bf6e152f98f5469d7"
server_key_version: "encryptionkey@0"
I20260620 09:16:50.756408 5706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:16:50.757073 5720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:50.757203 5706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:50.757280 5706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "44773593f63a4f68abd4b8442a50ab6d"
format_stamp: "Formatted at 2026-06-20 09:16:50 on dist-test-slave-rq8c"
server_key: "3b23c6e14dd827f849c955181d59c3a7"
server_key_iv: "c5ef028de48e419bf6e152f98f5469d7"
server_key_version: "encryptionkey@0"
I20260620 09:16:50.757337 5706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:50.765592 5706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:50.765832 5706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:50.765946 5706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:50.769404 5706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33761
I20260620 09:16:50.769443 5772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33761 every 8 connection(s)
I20260620 09:16:50.769724 5706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:50.770320 5773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:50.771144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 5706
I20260620 09:16:50.771242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:50.771397 392 external_mini_cluster.cc:1428] Setting key 1109eccb67f20dd263e37f323773e98d
I20260620 09:16:50.773211 5773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d: Bootstrap starting.
I20260620 09:16:50.773713 5773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:50.773953 5773 log.cc:824] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:50.774530 5773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d: No bootstrap required, opened a new log
I20260620 09:16:50.776221 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.771874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.0.98.62:33761)
0620 09:16:50.772049 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:50.772056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:50.772079 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:16:50.772459 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:50.772465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:50.772522 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:50.772630 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:50.772838 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.773737 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.774503 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.774939 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.775053 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:50.775675 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:50.775679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:50.775679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:50.775827 (+ 148us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:50.775858 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:50.775868 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:50.775918 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:50.775992 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 09:16:50.776063 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260620 09:16:50.776266 5773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44773593f63a4f68abd4b8442a50ab6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33761 } }
I20260620 09:16:50.776432 5773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:50.776464 5773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44773593f63a4f68abd4b8442a50ab6d, State: Initialized, Role: FOLLOWER
I20260620 09:16:50.776577 5773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44773593f63a4f68abd4b8442a50ab6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33761 } }
I20260620 09:16:50.776634 5773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:50.776662 5773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:50.776708 5773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:50.776970 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:16:50.777215 5773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44773593f63a4f68abd4b8442a50ab6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33761 } }
I20260620 09:16:50.777302 5773 leader_election.cc:302] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44773593f63a4f68abd4b8442a50ab6d; no voters:
I20260620 09:16:50.777436 5773 leader_election.cc:288] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:50.777472 5778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:50.777572 5778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [term 1 LEADER]: Becoming Leader. State: Replica: 44773593f63a4f68abd4b8442a50ab6d, State: Running, Role: LEADER
I20260620 09:16:50.777660 5778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44773593f63a4f68abd4b8442a50ab6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33761 } }
I20260620 09:16:50.777698 5773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:50.778014 5778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44773593f63a4f68abd4b8442a50ab6d. Latest consensus state: current_term: 1 leader_uuid: "44773593f63a4f68abd4b8442a50ab6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44773593f63a4f68abd4b8442a50ab6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33761 } } }
I20260620 09:16:50.778100 5778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:50.778018 5779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44773593f63a4f68abd4b8442a50ab6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44773593f63a4f68abd4b8442a50ab6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33761 } } }
I20260620 09:16:50.778235 5779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:50.778283 5784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:50.778746 5784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:50.780097 5784 catalog_manager.cc:1347] Generated new cluster ID: 3b6ef8c278d54824bbfe7d42d782174c
I20260620 09:16:50.780144 5784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:16:50.792701 5784 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:50.793423 5784 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:50.803295 5784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44773593f63a4f68abd4b8442a50ab6d: Generated new TSK 0
I20260620 09:16:50.803476 5784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:16:50.816308 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.812754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.0.98.62:33761)
0620 09:16:50.812798 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:50.812799 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:50.812879 (+ 80us) server_negotiation.cc:407] Connection header received
0620 09:16:50.812931 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:50.812932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:50.812943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:50.813020 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:16:50.813103 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.813860 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.814634 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.814958 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.815022 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:50.815170 (+ 148us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:16:50.816103 (+ 933us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35843/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 09:16:50.816232 (+ 129us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35843/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 09:16:50.816360 5776 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35843/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 09:16:50.818214 5802 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:16:50.812675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33761 (local address 127.0.0.1:45002)
0620 09:16:50.812805 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:50.812814 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:50.812872 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:50.813017 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:50.813017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:50.813019 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:16:50.813066 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:50.813067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.813955 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.813956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:50.814565 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:50.814567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.815031 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.815034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:50.815081 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:50.815086 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:16:50.816202 (+ 1116us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35843/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 09:16:50.818118 (+ 1916us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33761: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35843/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":1740,"thread_start_us":78,"threads_started":1}
I20260620 09:16:50.818531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 5706
E20260620 09:16:50.827203 5803 webserver.cc:533] Webserver: sslize failed:
E20260620 09:16:50.827265 5803 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T09:16:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (194 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T09:16:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:50Z Disabled control of system clock
I20260620 09:16:50.861336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41259
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41259
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/testchainca.cert with env {}
W20260620 09:16:50.939640 5807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:50.939914 5807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:50.939958 5807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:50.941123 5807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:50.941164 5807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:50.941179 5807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:50.941195 5807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:50.941210 5807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:16:50.941229 5807 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:16:50.941249 5807 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 09:16:50.942727 5807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:50.942973 5807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:50.943164 5807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:50.947980 5815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:50.948014 5813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:50.948010 5812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:50.948063 5807 server_base.cc:1034] running on GCE node
I20260620 09:16:50.948469 5807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:50.948747 5807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:50.949892 5807 hybrid_clock.cc:648] HybridClock initialized: now 1781947010949875 us; error 38 us; skew 500 ppm
I20260620 09:16:50.951131 5807 webserver.cc:460] Webserver started at http://127.0.98.62:42159/ using document root <none> and password file <none>
I20260620 09:16:50.951345 5807 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:50.951387 5807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:50.951500 5807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:50.952575 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "8327d95932e24cb88c43780fa309e324"
format_stamp: "Formatted at 2026-06-20 09:16:50 on dist-test-slave-rq8c"
server_key: "e35d07211ee65a4ec25ceaaf90c458fb"
server_key_iv: "3b2c3d42fcd930d2fe34e37c4e6a3fed"
server_key_version: "encryptionkey@0"
I20260620 09:16:50.952894 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "8327d95932e24cb88c43780fa309e324"
format_stamp: "Formatted at 2026-06-20 09:16:50 on dist-test-slave-rq8c"
server_key: "e35d07211ee65a4ec25ceaaf90c458fb"
server_key_iv: "3b2c3d42fcd930d2fe34e37c4e6a3fed"
server_key_version: "encryptionkey@0"
I20260620 09:16:50.954087 5807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:50.954825 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:50.955031 5807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:50.955119 5807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "8327d95932e24cb88c43780fa309e324"
format_stamp: "Formatted at 2026-06-20 09:16:50 on dist-test-slave-rq8c"
server_key: "e35d07211ee65a4ec25ceaaf90c458fb"
server_key_iv: "3b2c3d42fcd930d2fe34e37c4e6a3fed"
server_key_version: "encryptionkey@0"
I20260620 09:16:50.955188 5807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:50.969276 5807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:50.969554 5807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:50.969674 5807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:50.973142 5807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41259
I20260620 09:16:50.973179 5873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41259 every 8 connection(s)
I20260620 09:16:50.973479 5807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:50.974370 5874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:50.974812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 5807
I20260620 09:16:50.974910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:50.975095 392 external_mini_cluster.cc:1428] Setting key c9772d0b34cc7064e876c085baee72d1
I20260620 09:16:50.976861 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324: Bootstrap starting.
I20260620 09:16:50.977408 5874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:50.977670 5874 log.cc:824] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:50.978253 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324: No bootstrap required, opened a new log
I20260620 09:16:50.980068 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:50.975582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.0.98.62:41259)
0620 09:16:50.975788 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:50.975796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:50.975821 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:16:50.976218 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:50.976224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:50.976282 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:50.976407 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:50.976597 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.977500 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.978285 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:50.978734 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:50.978842 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:50.979465 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:50.979470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:50.979470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:16:50.979660 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:50.979703 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:16:50.979715 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:50.979767 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:50.979833 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 09:16:50.979906 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260620 09:16:50.980024 5874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8327d95932e24cb88c43780fa309e324" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41259 } }
I20260620 09:16:50.980166 5874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:50.980213 5874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8327d95932e24cb88c43780fa309e324, State: Initialized, Role: FOLLOWER
I20260620 09:16:50.980474 5874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8327d95932e24cb88c43780fa309e324" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41259 } }
I20260620 09:16:50.980557 5874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:50.980598 5874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:50.980646 5874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:50.980863 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:16:50.981242 5874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8327d95932e24cb88c43780fa309e324" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41259 } }
I20260620 09:16:50.981335 5874 leader_election.cc:302] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8327d95932e24cb88c43780fa309e324; no voters:
I20260620 09:16:50.981498 5874 leader_election.cc:288] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:50.981534 5879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:50.981789 5879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [term 1 LEADER]: Becoming Leader. State: Replica: 8327d95932e24cb88c43780fa309e324, State: Running, Role: LEADER
I20260620 09:16:50.981866 5879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8327d95932e24cb88c43780fa309e324" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41259 } }
I20260620 09:16:50.981909 5874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:50.982182 5879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8327d95932e24cb88c43780fa309e324. Latest consensus state: current_term: 1 leader_uuid: "8327d95932e24cb88c43780fa309e324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8327d95932e24cb88c43780fa309e324" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41259 } } }
I20260620 09:16:50.982219 5881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8327d95932e24cb88c43780fa309e324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8327d95932e24cb88c43780fa309e324" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41259 } } }
I20260620 09:16:50.982275 5879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:50.982295 5881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:50.982478 5885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:50.983119 5885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:50.984571 5885 catalog_manager.cc:1347] Generated new cluster ID: 3eedcc02a996483fada5d7cc0f35c97a
I20260620 09:16:50.984622 5885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:16:50.995002 5885 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:50.995699 5885 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:51.000705 5885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8327d95932e24cb88c43780fa309e324: Generated new TSK 0
I20260620 09:16:51.000869 5885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:16:51.029621 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:51.011879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.0.98.62:41259)
0620 09:16:51.011926 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:51.011927 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:51.011994 (+ 67us) server_negotiation.cc:407] Connection header received
0620 09:16:51.012072 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:51.012072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:51.012083 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:51.012138 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:16:51.012263 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:51.012763 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:51.013848 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:51.014061 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:51.014118 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:51.014274 (+ 156us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:16:51.029417 (+ 15143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 09:16:51.029563 (+ 146us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42658: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 09:16:51.029680 5877 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42658: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 09:16:51.029780 5903 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:16:51.011796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41259 (local address 127.0.0.1:42658)
0620 09:16:51.011929 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:51.011940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:51.012008 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:51.012152 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:51.012153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:51.012155 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:16:51.012209 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:51.012211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:51.012826 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:51.012826 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:51.013759 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:51.013761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:51.014132 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:51.014135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:51.014193 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:51.014198 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:16:51.029545 (+ 15347us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 09:16:51.029674 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41259: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260620 09:16:51.030084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 5807
2026-06-20T09:16:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (191 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:16:51 dist-test-slave-rq8c krb5kdc[5915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:16:51 dist-test-slave-rq8c krb5kdc[5915](info): set up 2 sockets
Jun 20 09:16:51 dist-test-slave-rq8c krb5kdc[5915](info): commencing operation
krb5kdc: starting...
W20260620 09:16:52.143473 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:16:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:16:52.245977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36205
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:52.322491 5928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:52.322695 5928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:52.322723 5928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:52.323866 5928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:52.323904 5928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:52.323922 5928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:52.323948 5928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:52.323969 5928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:52.325474 5928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:52.325707 5928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:52.325898 5928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:52.330282 5933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:52.330375 5928 server_base.cc:1034] running on GCE node
W20260620 09:16:52.330269 5936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:52.330268 5934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:52.330739 5928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:52.330976 5928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:52.332103 5928 hybrid_clock.cc:648] HybridClock initialized: now 1781947012332089 us; error 28 us; skew 500 ppm
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:52.334424 5928 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:16:52.334816 5928 webserver.cc:460] Webserver started at http://127.0.98.62:39871/ using document root <none> and password file <none>
I20260620 09:16:52.335014 5928 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:52.335060 5928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:52.335182 5928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:52.336228 5928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "05cd2b2a89e241689a42944343d36e40"
format_stamp: "Formatted at 2026-06-20 09:16:52 on dist-test-slave-rq8c"
server_key: "2003e83db65ac78252260680373d2707"
server_key_iv: "2a514e406feb3d73470869f084132eea"
server_key_version: "encryptionkey@0"
I20260620 09:16:52.336551 5928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "05cd2b2a89e241689a42944343d36e40"
format_stamp: "Formatted at 2026-06-20 09:16:52 on dist-test-slave-rq8c"
server_key: "2003e83db65ac78252260680373d2707"
server_key_iv: "2a514e406feb3d73470869f084132eea"
server_key_version: "encryptionkey@0"
I20260620 09:16:52.337707 5928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:52.338331 5943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:52.338475 5928 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:52.338554 5928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "05cd2b2a89e241689a42944343d36e40"
format_stamp: "Formatted at 2026-06-20 09:16:52 on dist-test-slave-rq8c"
server_key: "2003e83db65ac78252260680373d2707"
server_key_iv: "2a514e406feb3d73470869f084132eea"
server_key_version: "encryptionkey@0"
I20260620 09:16:52.338613 5928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:52.364071 5928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:52.365787 5928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:52.365916 5928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:52.370127 5928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36205
I20260620 09:16:52.370164 5995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36205 every 8 connection(s)
I20260620 09:16:52.370621 5928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:52.371218 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:52.373335 5996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40: Bootstrap starting.
I20260620 09:16:52.373840 5996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:52.374060 5996 log.cc:824] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:52.374615 5996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40: No bootstrap required, opened a new log
I20260620 09:16:52.376125 5996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cd2b2a89e241689a42944343d36e40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36205 } }
I20260620 09:16:52.376252 5996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:52.376281 5996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05cd2b2a89e241689a42944343d36e40, State: Initialized, Role: FOLLOWER
I20260620 09:16:52.376398 5996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cd2b2a89e241689a42944343d36e40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36205 } }
I20260620 09:16:52.376477 5996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:52.376505 5996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:52.376535 5996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:52.377290 5996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cd2b2a89e241689a42944343d36e40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36205 } }
I20260620 09:16:52.377378 5996 leader_election.cc:302] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05cd2b2a89e241689a42944343d36e40; no voters:
I20260620 09:16:52.377504 5996 leader_election.cc:288] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:52.377552 5999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:52.377704 5996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:52.377717 5999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [term 1 LEADER]: Becoming Leader. State: Replica: 05cd2b2a89e241689a42944343d36e40, State: Running, Role: LEADER
I20260620 09:16:52.377872 5999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cd2b2a89e241689a42944343d36e40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36205 } }
I20260620 09:16:52.378223 6001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05cd2b2a89e241689a42944343d36e40. Latest consensus state: current_term: 1 leader_uuid: "05cd2b2a89e241689a42944343d36e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cd2b2a89e241689a42944343d36e40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36205 } } }
I20260620 09:16:52.378341 6001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:52.378443 6000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05cd2b2a89e241689a42944343d36e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05cd2b2a89e241689a42944343d36e40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36205 } } }
I20260620 09:16:52.378579 6000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:52.378755 6008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:52.379215 6008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:52.380095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 5928
I20260620 09:16:52.380182 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:52.380350 392 external_mini_cluster.cc:1428] Setting key 0a29c2179c70eda8780c2caa1d170d2d
I20260620 09:16:52.380748 6008 catalog_manager.cc:1347] Generated new cluster ID: 40eb3bb775a340d5a6a5f26dba3297b1
I20260620 09:16:52.380798 6008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:52.391832 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:52.380945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.0.98.62:36205)
0620 09:16:52.381136 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:52.381145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:52.381179 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:16:52.381290 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:52.381297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:52.381361 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:52.381512 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:52.382355 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.383878 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.384863 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.385511 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.385622 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:52.388455 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:52.388459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:52.388460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:52.388473 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:52.390298 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.390668 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.390671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.390671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.390763 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.390935 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.390936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.390936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.391217 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:52.391356 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:52.391448 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 09:16:52.391571 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
I20260620 09:16:52.426259 6008 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:52.427018 6008 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:52.430541 6008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05cd2b2a89e241689a42944343d36e40: Generated new TSK 0
I20260620 09:16:52.430719 6008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:16:52.460310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36205
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:52.537746 6022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:52.537951 6022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:52.537982 6022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:52.539108 6022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:52.539144 6022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:52.539178 6022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:52.540639 6022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:52.540863 6022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:52.541079 6022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:52.545861 6027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:52.545923 6028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:52.545975 6030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:52.546000 6022 server_base.cc:1034] running on GCE node
I20260620 09:16:52.546277 6022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:52.546510 6022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:52.547667 6022 hybrid_clock.cc:648] HybridClock initialized: now 1781947012547646 us; error 28 us; skew 500 ppm
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:52.550012 6022 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:16:52.550417 6022 webserver.cc:460] Webserver started at http://127.0.98.1:33251/ using document root <none> and password file <none>
I20260620 09:16:52.550591 6022 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:52.550629 6022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:52.550716 6022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:52.551725 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "04564b2d359b4ec3adae86caf05634e7"
format_stamp: "Formatted at 2026-06-20 09:16:52 on dist-test-slave-rq8c"
server_key: "34fda71c87e382ff51499e615cf64efb"
server_key_iv: "b8fc540eb2368c1074f879a3495cf1c5"
server_key_version: "encryptionkey@0"
I20260620 09:16:52.552030 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "04564b2d359b4ec3adae86caf05634e7"
format_stamp: "Formatted at 2026-06-20 09:16:52 on dist-test-slave-rq8c"
server_key: "34fda71c87e382ff51499e615cf64efb"
server_key_iv: "b8fc540eb2368c1074f879a3495cf1c5"
server_key_version: "encryptionkey@0"
I20260620 09:16:52.553141 6022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:52.553741 6037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:52.553901 6022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:52.553975 6022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "04564b2d359b4ec3adae86caf05634e7"
format_stamp: "Formatted at 2026-06-20 09:16:52 on dist-test-slave-rq8c"
server_key: "34fda71c87e382ff51499e615cf64efb"
server_key_iv: "b8fc540eb2368c1074f879a3495cf1c5"
server_key_version: "encryptionkey@0"
I20260620 09:16:52.554026 6022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:52.568250 6022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:52.569962 6022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:52.570066 6022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:52.570235 6022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:52.570533 6022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:52.570570 6022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:52.570591 6022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:52.570614 6022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:52.575956 6022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35045
I20260620 09:16:52.576006 6150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35045 every 8 connection(s)
I20260620 09:16:52.576298 6022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:52.584879 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:52.576964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52265 (local address 127.0.98.62:36205)
0620 09:16:52.577014 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:52.577016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:52.577209 (+ 193us) server_negotiation.cc:407] Connection header received
0620 09:16:52.578231 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:52.578232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:52.578242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:52.578282 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:52.578904 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.579455 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.580443 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.580643 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.580693 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:52.582593 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:52.582594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:52.582595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:52.582601 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:52.583694 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.584045 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.584046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.584047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.584108 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.584335 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.584335 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.584336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.584447 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:52.584500 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:52.584740 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 09:16:52.584809 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:16:52.584954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 6022
I20260620 09:16:52.585036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:16:52.585037 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:52.576952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36205 (local address 127.0.98.1:52265)
0620 09:16:52.577106 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:52.577141 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:52.578130 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:52.578349 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:52.578352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:52.578602 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:52.578815 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:52.578820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.579526 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.579527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:52.580366 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:52.580368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.580699 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.580699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:52.580790 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:52.581226 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:52.581233 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:52.582503 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:52.583752 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:52.583757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:52.583759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:52.583963 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:52.584184 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:52.584184 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:52.584185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:52.584283 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:52.584496 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:52.584499 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:52.584612 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 09:16:52.584688 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:16:52.584815 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260620 09:16:52.585243 392 external_mini_cluster.cc:1428] Setting key 1ed78d36adc9a8d57b63b44b76dc64d1
I20260620 09:16:52.585456 6151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36205
I20260620 09:16:52.585543 6151 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:52.585716 6151 heartbeater.cc:511] Master 127.0.98.62:36205 requested a full tablet report, sending...
I20260620 09:16:52.586184 5960 ts_manager.cc:194] Registered new tserver with Master: 04564b2d359b4ec3adae86caf05634e7 (127.0.98.1:35045)
I20260620 09:16:52.586758 5960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52265
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:16:52.606189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36205
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:52.684495 6156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:52.684679 6156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:52.684708 6156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:52.685797 6156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:52.685834 6156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:52.685868 6156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:52.687285 6156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:52.687510 6156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:52.687737 6156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:52.692598 6164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:52.692631 6161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:52.692665 6156 server_base.cc:1034] running on GCE node
W20260620 09:16:52.692633 6162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:52.693037 6156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:52.693292 6156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:52.694456 6156 hybrid_clock.cc:648] HybridClock initialized: now 1781947012694443 us; error 32 us; skew 500 ppm
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:52.696857 6156 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:16:52.697244 6156 webserver.cc:460] Webserver started at http://127.0.98.2:35681/ using document root <none> and password file <none>
I20260620 09:16:52.697458 6156 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:52.697505 6156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:52.697623 6156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:52.698596 6156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "8344c56ec8ae410eacaefa84c24d0c51"
format_stamp: "Formatted at 2026-06-20 09:16:52 on dist-test-slave-rq8c"
server_key: "f2c49a084a0ee25be906f09b45fc5960"
server_key_iv: "1b1a4d1db9efd47da38720570c02f3eb"
server_key_version: "encryptionkey@0"
I20260620 09:16:52.698928 6156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8344c56ec8ae410eacaefa84c24d0c51"
format_stamp: "Formatted at 2026-06-20 09:16:52 on dist-test-slave-rq8c"
server_key: "f2c49a084a0ee25be906f09b45fc5960"
server_key_iv: "1b1a4d1db9efd47da38720570c02f3eb"
server_key_version: "encryptionkey@0"
I20260620 09:16:52.700217 6156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:52.700883 6171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:52.701031 6156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:52.701108 6156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "8344c56ec8ae410eacaefa84c24d0c51"
format_stamp: "Formatted at 2026-06-20 09:16:52 on dist-test-slave-rq8c"
server_key: "f2c49a084a0ee25be906f09b45fc5960"
server_key_iv: "1b1a4d1db9efd47da38720570c02f3eb"
server_key_version: "encryptionkey@0"
I20260620 09:16:52.701169 6156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:52.713973 6156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:52.715809 6156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:52.715946 6156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:52.716150 6156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:52.716481 6156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:52.716516 6156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:52.716543 6156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:52.716560 6156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:52.721864 6156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38471
I20260620 09:16:52.721928 6284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38471 every 8 connection(s)
I20260620 09:16:52.722204 6156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:52.730383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 6156
I20260620 09:16:52.730476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:16:52.730664 392 external_mini_cluster.cc:1428] Setting key d8eeb0226024c871c32cdab16fd6734a
I20260620 09:16:52.730976 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:52.722878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44281 (local address 127.0.98.62:36205)
0620 09:16:52.722944 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:52.722946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:52.723144 (+ 198us) server_negotiation.cc:407] Connection header received
0620 09:16:52.724101 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:52.724102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:52.724111 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:52.724163 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:52.724765 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.725268 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.726239 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.726427 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.726482 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:52.728543 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:52.728544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:52.728544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:52.728551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:52.729760 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.730067 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.730067 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.730068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.730109 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.730295 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.730295 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.730296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.730440 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:52.730508 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:52.730780 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 09:16:52.730890 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:16:52.731175 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:52.722870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36205 (local address 127.0.98.2:44281)
0620 09:16:52.723029 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:52.723064 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:52.723993 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:52.724220 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:52.724223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:52.724472 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:52.724662 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:52.724666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.725358 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.725358 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:52.726155 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:52.726157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.726518 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.726519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:52.726634 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:52.727109 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:52.727117 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:52.728460 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:52.729821 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:52.729824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:52.729826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:52.730019 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:52.730152 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:52.730153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:52.730153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:52.730243 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:52.730522 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:52.730527 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:52.730655 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 09:16:52.730754 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 09:16:52.730909 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260620 09:16:52.731753 6285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36205
I20260620 09:16:52.731858 6285 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:52.732038 6285 heartbeater.cc:511] Master 127.0.98.62:36205 requested a full tablet report, sending...
I20260620 09:16:52.732410 5960 ts_manager.cc:194] Registered new tserver with Master: 8344c56ec8ae410eacaefa84c24d0c51 (127.0.98.2:38471)
I20260620 09:16:52.732800 5960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44281
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:16:52.752002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36205
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:52.828845 6290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:52.829034 6290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:52.829056 6290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:52.830134 6290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:52.830165 6290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:52.830189 6290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:52.831571 6290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:52.831779 6290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:52.831961 6290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:52.836732 6298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:52.836755 6295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:52.836807 6290 server_base.cc:1034] running on GCE node
W20260620 09:16:52.836764 6296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:52.837095 6290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:52.837292 6290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:52.838433 6290 hybrid_clock.cc:648] HybridClock initialized: now 1781947012838423 us; error 30 us; skew 500 ppm
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:52.840793 6290 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:16:52.841212 6290 webserver.cc:460] Webserver started at http://127.0.98.3:46007/ using document root <none> and password file <none>
I20260620 09:16:52.841434 6290 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:52.841482 6290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:52.841604 6290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:52.842590 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "5a7f74a30dba4837a23b12a249e914c5"
format_stamp: "Formatted at 2026-06-20 09:16:52 on dist-test-slave-rq8c"
server_key: "2532b64418f9276f6ba1155fc3535fac"
server_key_iv: "a8cbd87628c1098992687f54cee66737"
server_key_version: "encryptionkey@0"
I20260620 09:16:52.842898 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5a7f74a30dba4837a23b12a249e914c5"
format_stamp: "Formatted at 2026-06-20 09:16:52 on dist-test-slave-rq8c"
server_key: "2532b64418f9276f6ba1155fc3535fac"
server_key_iv: "a8cbd87628c1098992687f54cee66737"
server_key_version: "encryptionkey@0"
I20260620 09:16:52.844131 6290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:52.844761 6305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:52.844897 6290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:52.844976 6290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "5a7f74a30dba4837a23b12a249e914c5"
format_stamp: "Formatted at 2026-06-20 09:16:52 on dist-test-slave-rq8c"
server_key: "2532b64418f9276f6ba1155fc3535fac"
server_key_iv: "a8cbd87628c1098992687f54cee66737"
server_key_version: "encryptionkey@0"
I20260620 09:16:52.845033 6290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:52.851063 6290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:52.853027 6290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:52.853154 6290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:52.853336 6290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:52.853603 6290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:52.853636 6290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:52.853660 6290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:52.853684 6290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:52.859212 6290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40663
I20260620 09:16:52.859246 6418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40663 every 8 connection(s)
I20260620 09:16:52.859537 6290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:16:52.866022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 6290
I20260620 09:16:52.866109 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:16:52.866268 392 external_mini_cluster.cc:1428] Setting key 0f189c6e32d30d45418b3f75e9797586
I20260620 09:16:52.868419 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:52.860225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39809 (local address 127.0.98.62:36205)
0620 09:16:52.860289 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:52.860291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:52.860494 (+ 203us) server_negotiation.cc:407] Connection header received
0620 09:16:52.861411 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:52.861412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:52.861425 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:52.861475 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:52.862066 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.862563 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.863535 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.863811 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.863879 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:52.865834 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:52.865836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:52.865836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:52.865844 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:52.867347 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.867728 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.867729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.867729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.867776 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.867954 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.867954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.867955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.868049 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:52.868092 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:52.868293 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 09:16:52.868355 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:16:52.868613 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:52.860208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36205 (local address 127.0.98.3:39809)
0620 09:16:52.860372 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:52.860407 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:52.861303 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:52.861545 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:52.861548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:52.861785 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:52.861974 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:52.861979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.862643 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.862644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:52.863449 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:52.863452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.863887 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.863888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:52.864018 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:52.864504 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:52.864513 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:52.865742 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:52.867421 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:52.867427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:52.867429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:52.867669 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:52.867819 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:52.867820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:52.867820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:52.867910 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:52.868089 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:52.868092 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:52.868202 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 09:16:52.868275 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:16:52.868391 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260620 09:16:52.869007 6419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36205
I20260620 09:16:52.869098 6419 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:52.869263 6419 heartbeater.cc:511] Master 127.0.98.62:36205 requested a full tablet report, sending...
I20260620 09:16:52.869603 5960 ts_manager.cc:194] Registered new tserver with Master: 5a7f74a30dba4837a23b12a249e914c5 (127.0.98.3:40663)
I20260620 09:16:52.869976 5960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39809
I20260620 09:16:52.877915 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/test-admin.keytab.
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:52.891528 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 09:16:52.893249 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:16:52.892479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.0.98.1:35045)
0620 09:16:52.892638 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:52.892642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:52.892653 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:16:52.892679 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:52.892681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:52.892695 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:52.892759 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:52.893166 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56734: BlockingRecv error: recv got EOF from 127.0.0.1:56734 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
W20260620 09:16:52.893399 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/krb5cc)
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:52.894337 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:52.895155 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:16:52.902272 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:52.895689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.0.98.1:35045)
0620 09:16:52.895745 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:52.895751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:52.895777 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:16:52.895848 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:52.895849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:52.895860 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:52.895909 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:52.896248 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.896961 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.897655 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.898015 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.898090 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:52.899959 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:52.899967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:52.899967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:52.899974 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:52.901431 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.901676 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.901679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.901679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.901727 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.901842 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.901842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.901842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.901959 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:52.902053 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:52.902102 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 09:16:52.902158 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 09:16:52.904036 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:16:52.903476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.0.98.1:35045)
0620 09:16:52.903511 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:52.903513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:52.903583 (+ 70us) server_negotiation.cc:407] Connection header received
0620 09:16:52.903669 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:52.903669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:52.903680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:52.903736 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:52.904000 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56740: BlockingRecv error: recv got EOF from 127.0.0.1:56740 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 09:16:52.904145 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:52.905071 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:52.905809 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:16:52.911942 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:52.906292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.0.98.1:35045)
0620 09:16:52.906357 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:52.906359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:52.906367 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:52.906420 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:52.906420 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:52.906429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:52.906474 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:52.906801 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.907305 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.908002 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.908204 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.908263 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:52.909976 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:52.909977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:52.909977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:52.909983 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:52.911151 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.911388 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.911389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.911389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.911431 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.911545 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.911545 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.911546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.911649 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:52.911720 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:52.911803 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 09:16:52.911859 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 09:16:52.913272 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:16:52.912787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.0.98.1:35045)
0620 09:16:52.912831 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:52.912833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:52.912898 (+ 65us) server_negotiation.cc:407] Connection header received
0620 09:16:52.912953 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:52.912954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:52.912963 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:52.913005 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:52.913228 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56750: BlockingRecv error: recv got EOF from 127.0.0.1:56750 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 09:16:52.913335 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946986834672-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:52.914156 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:52.914835 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:16:52.920759 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:52.915280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.0.98.1:35045)
0620 09:16:52.915323 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:52.915324 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:52.915347 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:16:52.915409 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:52.915410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:52.915418 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:52.915458 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:52.915811 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.916272 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.916914 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:52.917103 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:52.917154 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:52.918811 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:52.918813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:52.918813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:52.918819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:52.919879 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.920145 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.920146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.920146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.920192 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:52.920341 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:52.920343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:52.920343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:52.920434 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:52.920512 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:52.920610 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 09:16:52.920678 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:16:52.921406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 6022
I20260620 09:16:52.926209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 6156
I20260620 09:16:52.930032 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 6290
I20260620 09:16:52.933640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 5928
2026-06-20T09:16:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1735 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[6439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[6439](info): set up 2 sockets
Jun 20 09:16:52 dist-test-slave-rq8c krb5kdc[6439](info): commencing operation
krb5kdc: starting...
W20260620 09:16:53.879520 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:16:53 dist-test-slave-rq8c krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:16:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 09:16:53.996295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41791
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:54.074838 6452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:54.075045 6452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:54.075066 6452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:54.076251 6452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:54.076292 6452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:54.076305 6452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:54.076316 6452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:54.076327 6452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:54.077795 6452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:54.078017 6452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:54.078225 6452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:54.082864 6458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:54.082927 6452 server_base.cc:1034] running on GCE node
W20260620 09:16:54.083027 6460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:54.083027 6457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:54.083362 6452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:54.083765 6452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:54.084955 6452 hybrid_clock.cc:648] HybridClock initialized: now 1781947014084943 us; error 29 us; skew 500 ppm
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:54.087447 6452 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 09:16:54.087890 6452 webserver.cc:460] Webserver started at http://127.0.98.62:43965/ using document root <none> and password file <none>
I20260620 09:16:54.088068 6452 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:54.088109 6452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:54.088207 6452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:54.089196 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "abd5a2ea27e2141c7b61ec550495c077"
server_key_iv: "c785c69b696110100d9245a86a3a6653"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.089483 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "abd5a2ea27e2141c7b61ec550495c077"
server_key_iv: "c785c69b696110100d9245a86a3a6653"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.090673 6452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:54.091368 6467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.091521 6452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:54.091602 6452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "abd5a2ea27e2141c7b61ec550495c077"
server_key_iv: "c785c69b696110100d9245a86a3a6653"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.091687 6452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:54.102380 6452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:54.104473 6452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:54.104579 6452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:54.108170 6452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41791
I20260620 09:16:54.108210 6519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41791 every 8 connection(s)
I20260620 09:16:54.108517 6452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:54.109431 6520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:54.110582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 6452
I20260620 09:16:54.110674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:54.110847 392 external_mini_cluster.cc:1428] Setting key 81ff88c00dc83e36514bc67f2ebfea5d
I20260620 09:16:54.111929 6520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca: Bootstrap starting.
I20260620 09:16:54.112469 6520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:54.112712 6520 log.cc:824] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:54.113317 6520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca: No bootstrap required, opened a new log
I20260620 09:16:54.114984 6520 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } }
I20260620 09:16:54.115113 6520 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:54.115142 6520 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bd2c7b810f4490a53b96a66bd7a3ca, State: Initialized, Role: FOLLOWER
I20260620 09:16:54.115262 6520 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } }
I20260620 09:16:54.115330 6520 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:54.115368 6520 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:54.115412 6520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:54.115957 6520 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } }
I20260620 09:16:54.116044 6520 leader_election.cc:302] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4bd2c7b810f4490a53b96a66bd7a3ca; no voters:
I20260620 09:16:54.116184 6520 leader_election.cc:288] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:54.116214 6525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:54.116331 6525 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 1 LEADER]: Becoming Leader. State: Replica: a4bd2c7b810f4490a53b96a66bd7a3ca, State: Running, Role: LEADER
I20260620 09:16:54.116456 6520 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:54.116425 6525 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } }
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:54.116811 6527 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4bd2c7b810f4490a53b96a66bd7a3ca. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } }
I20260620 09:16:54.116885 6527 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:54.117022 6526 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } }
I20260620 09:16:54.117067 6526 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:54.117390 6531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:54.117884 6531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:54.119169 6531 catalog_manager.cc:1347] Generated new cluster ID: dbf97b58aadf4ef59ca61a172f58a443
I20260620 09:16:54.119216 6531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:16:54.120594 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.111332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.0.98.62:41791)
0620 09:16:54.111505 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:54.111513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:54.111536 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:16:54.111952 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:54.111959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:54.112008 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:54.112161 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:54.112756 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.113720 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.114419 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.114808 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.114899 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.117296 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:54.117300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:54.117300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:54.117312 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:54.119073 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.119416 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.119419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.119419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.119531 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.119766 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.119767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.119767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.120040 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:54.120164 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:54.120285 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 09:16:54.120411 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260620 09:16:54.135324 6531 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:54.135886 6531 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:54.145581 6531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca: Generated new TSK 0
I20260620 09:16:54.145757 6531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 09:16:54.171540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41791
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:54.249871 6546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:54.250078 6546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:54.250108 6546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:54.251257 6546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:54.251297 6546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:54.251333 6546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:54.252837 6546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:54.253062 6546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:54.253266 6546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:54.258019 6551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:54.257970 6554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:54.257980 6552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:54.258149 6546 server_base.cc:1034] running on GCE node
I20260620 09:16:54.258352 6546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:54.258541 6546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:54.259673 6546 hybrid_clock.cc:648] HybridClock initialized: now 1781947014259658 us; error 28 us; skew 500 ppm
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:54.264875 6546 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 09:16:54.265280 6546 webserver.cc:460] Webserver started at http://127.0.98.1:35061/ using document root <none> and password file <none>
I20260620 09:16:54.265486 6546 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:54.265532 6546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:54.265652 6546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:54.266660 6546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "8587525e365e445c8df92975c7dbd21c"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "255ded57396df5fa70b04f808ae9ed8e"
server_key_iv: "3072f3b08b57126d2879819233476cc1"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.266983 6546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8587525e365e445c8df92975c7dbd21c"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "255ded57396df5fa70b04f808ae9ed8e"
server_key_iv: "3072f3b08b57126d2879819233476cc1"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.268208 6546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:54.268839 6561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.268978 6546 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:54.269057 6546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "8587525e365e445c8df92975c7dbd21c"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "255ded57396df5fa70b04f808ae9ed8e"
server_key_iv: "3072f3b08b57126d2879819233476cc1"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.269111 6546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:54.280897 6546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:54.283018 6546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:54.283155 6546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:54.283391 6546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:54.283788 6546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:54.283828 6546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.283847 6546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:54.283859 6546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.289214 6546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46105
I20260620 09:16:54.289286 6674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46105 every 8 connection(s)
I20260620 09:16:54.289547 6546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:54.295881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 6546
I20260620 09:16:54.295980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:16:54.296172 392 external_mini_cluster.cc:1428] Setting key 0f77c77d1347dfd05a9a65aaa0c3c7a4
I20260620 09:16:54.298164 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.290199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45079 (local address 127.0.98.62:41791)
0620 09:16:54.290278 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:54.290280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:54.290457 (+ 177us) server_negotiation.cc:407] Connection header received
0620 09:16:54.290946 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:54.290947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:54.290958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:54.291010 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:54.291939 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.292427 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.293362 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.293556 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.293609 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.295593 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:54.295594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:54.295594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:54.295601 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:54.296981 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.297331 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.297332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.297333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.297389 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.297590 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.297591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.297591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.297697 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:54.297752 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:54.297998 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 09:16:54.298078 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:16:54.298362 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.290185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.1:45079)
0620 09:16:54.290342 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:54.290377 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:54.290851 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:54.291064 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:54.291067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:54.291605 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:54.291860 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.291865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.292493 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.292494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.293290 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.293292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.293615 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.293615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.293703 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.294178 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:54.294186 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:54.295511 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:54.297048 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.297054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.297056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.297256 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.297440 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.297441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.297441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.297536 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.297750 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:54.297756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:54.297884 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 09:16:54.297970 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:16:54.298116 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":96,"threads_started":1}
I20260620 09:16:54.298748 6675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41791
I20260620 09:16:54.298834 6675 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:54.298996 6675 heartbeater.cc:511] Master 127.0.98.62:41791 requested a full tablet report, sending...
I20260620 09:16:54.299513 6484 ts_manager.cc:194] Registered new tserver with Master: 8587525e365e445c8df92975c7dbd21c (127.0.98.1:46105)
I20260620 09:16:54.300462 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45079
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 09:16:54.316051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41791
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:54.393221 6680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:54.393411 6680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:54.393432 6680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:54.394524 6680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:54.394562 6680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:54.394586 6680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:54.396046 6680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:54.396248 6680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:54.396438 6680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:54.401124 6686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:54.401151 6685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:54.401259 6680 server_base.cc:1034] running on GCE node
W20260620 09:16:54.401124 6688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:54.401502 6680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:54.401717 6680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:54.402843 6680 hybrid_clock.cc:648] HybridClock initialized: now 1781947014402834 us; error 27 us; skew 500 ppm
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:54.405167 6680 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 09:16:54.405578 6680 webserver.cc:460] Webserver started at http://127.0.98.2:44057/ using document root <none> and password file <none>
I20260620 09:16:54.405764 6680 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:54.405809 6680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:54.405932 6680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:54.406935 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "fbc5044d7eec4e3db65214b485134ca8"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "ada662b47bb91121c245589b754eea69"
server_key_iv: "97d89e14fc464d6abfef5fbad89f9a83"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.407279 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fbc5044d7eec4e3db65214b485134ca8"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "ada662b47bb91121c245589b754eea69"
server_key_iv: "97d89e14fc464d6abfef5fbad89f9a83"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.408530 6680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:16:54.409191 6695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.409348 6680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:54.409423 6680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "fbc5044d7eec4e3db65214b485134ca8"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "ada662b47bb91121c245589b754eea69"
server_key_iv: "97d89e14fc464d6abfef5fbad89f9a83"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.409482 6680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:54.428850 6680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:54.430943 6680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:54.431072 6680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:54.431288 6680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:54.431561 6680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:54.431593 6680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.431641 6680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:54.431660 6680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.436897 6680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40687
I20260620 09:16:54.436949 6808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40687 every 8 connection(s)
I20260620 09:16:54.437234 6680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:16:54.440250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 6680
I20260620 09:16:54.440371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:16:54.440542 392 external_mini_cluster.cc:1428] Setting key 878c489e51933b0be86f72b15f64c043
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:54.446499 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.437882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34549 (local address 127.0.98.62:41791)
0620 09:16:54.437931 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:54.437933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:54.438105 (+ 172us) server_negotiation.cc:407] Connection header received
0620 09:16:54.438635 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:54.438637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:54.438649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:54.438700 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:54.439593 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.440252 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.441406 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.441664 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.441737 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.443866 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:54.443868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:54.443868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:54.443876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:54.445311 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.445671 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.445672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.445673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.445733 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.445929 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.445930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.445930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.446058 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:54.446127 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:54.446333 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 09:16:54.446408 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:16:54.446570 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.437849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.2:34549)
0620 09:16:54.437981 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:54.438021 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:54.438529 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:54.438772 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:54.438777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:54.439305 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:54.439501 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.439506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.440350 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.440351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.441329 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.441331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.441739 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.441740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.441833 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.442296 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:54.442303 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:54.443786 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:54.445386 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.445392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.445393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.445606 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.445789 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.445789 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.445790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.445877 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.446115 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:54.446118 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:54.446223 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 09:16:54.446297 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:16:54.446407 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260620 09:16:54.447046 6809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41791
I20260620 09:16:54.447135 6809 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:54.447295 6809 heartbeater.cc:511] Master 127.0.98.62:41791 requested a full tablet report, sending...
I20260620 09:16:54.447683 6484 ts_manager.cc:194] Registered new tserver with Master: fbc5044d7eec4e3db65214b485134ca8 (127.0.98.2:40687)
I20260620 09:16:54.448028 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34549
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 09:16:54.461935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41791
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:54.539163 6814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:54.539358 6814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:54.539381 6814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:54.540596 6814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:54.540642 6814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:54.540668 6814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:54.542045 6814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:54.542261 6814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:54.542470 6814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:54.547436 6822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:54.547411 6820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:54.547472 6814 server_base.cc:1034] running on GCE node
W20260620 09:16:54.547432 6819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:54.547828 6814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:54.548060 6814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:54.549198 6814 hybrid_clock.cc:648] HybridClock initialized: now 1781947014549185 us; error 33 us; skew 500 ppm
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:54.551717 6814 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 09:16:54.552117 6814 webserver.cc:460] Webserver started at http://127.0.98.3:41673/ using document root <none> and password file <none>
I20260620 09:16:54.552297 6814 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:54.552336 6814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:54.552426 6814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:54.553445 6814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "ec6b0b7db96744d3919456e1c5f5cc39"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "092321c88e141d12061694e3fc57cc6f"
server_key_iv: "f3a4ac86189abb7df97c7d9f92c78ebe"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.553740 6814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ec6b0b7db96744d3919456e1c5f5cc39"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "092321c88e141d12061694e3fc57cc6f"
server_key_iv: "f3a4ac86189abb7df97c7d9f92c78ebe"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.554958 6814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:16:54.555668 6829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.555809 6814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:54.555891 6814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "ec6b0b7db96744d3919456e1c5f5cc39"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "092321c88e141d12061694e3fc57cc6f"
server_key_iv: "f3a4ac86189abb7df97c7d9f92c78ebe"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.555949 6814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:54.570111 6814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:54.572068 6814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:54.572172 6814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:54.572367 6814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:54.572620 6814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:54.572649 6814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.572670 6814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:54.572690 6814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.577992 6814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39167
I20260620 09:16:54.578078 6942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39167 every 8 connection(s)
I20260620 09:16:54.578346 6814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:54.586020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 6814
I20260620 09:16:54.586102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:16:54.586249 392 external_mini_cluster.cc:1428] Setting key 23090be2a43e37382c3cbec9d67de645
I20260620 09:16:54.587122 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.579075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41057 (local address 127.0.98.62:41791)
0620 09:16:54.579134 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:54.579136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:54.579359 (+ 223us) server_negotiation.cc:407] Connection header received
0620 09:16:54.579922 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:54.579923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:54.579935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:54.579986 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:54.580907 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.581400 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.582374 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.582569 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.582625 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.584676 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:54.584677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:54.584677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:54.584685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:54.585887 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.586264 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.586265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.586265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.586325 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.586535 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.586536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.586537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.586648 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:54.586704 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:54.586960 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 09:16:54.587037 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:16:54.587296 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.579039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.3:41057)
0620 09:16:54.579247 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:54.579283 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:54.579823 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:54.580058 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:54.580061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:54.580612 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:54.580810 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.580815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.581480 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.581481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.582284 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.582287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.582635 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.582636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.582753 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.583235 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:54.583243 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:54.584591 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:54.585957 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.585963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.585965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.586189 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.586372 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.586373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.586373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.586481 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.586701 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:54.586707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:54.586844 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 09:16:54.586931 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 09:16:54.587082 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":104,"threads_started":1}
I20260620 09:16:54.587671 6943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41791
I20260620 09:16:54.587769 6943 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:54.587939 6943 heartbeater.cc:511] Master 127.0.98.62:41791 requested a full tablet report, sending...
I20260620 09:16:54.588277 6484 ts_manager.cc:194] Registered new tserver with Master: ec6b0b7db96744d3919456e1c5f5cc39 (127.0.98.3:39167)
I20260620 09:16:54.588640 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41057
I20260620 09:16:54.597863 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 09:16:54.619341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37499
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41791,127.0.98.61:37499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:54.696748 6948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:54.696938 6948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:54.696957 6948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:54.698077 6948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:54.698118 6948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:54.698132 6948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:54.698143 6948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:54.698153 6948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:54.699592 6948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41791,127.0.98.61:37499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
W20260620 09:16:54.699719 6948 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:41791,127.0.98.61:37499'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 09:16:54.699859 6948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:54.700057 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:54.704388 6953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:54.704397 6954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:54.704413 6956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:54.704619 6948 server_base.cc:1034] running on GCE node
I20260620 09:16:54.704838 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:54.705111 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:54.706245 6948 hybrid_clock.cc:648] HybridClock initialized: now 1781947014706235 us; error 33 us; skew 500 ppm
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:54.708559 6948 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 09:16:54.708954 6948 webserver.cc:460] Webserver started at http://127.0.98.61:40667/ using document root <none> and password file <none>
I20260620 09:16:54.709136 6948 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:54.709178 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:54.709259 6948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:54.710219 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data/instance:
uuid: "04208839d9c74e7eb1b9db21c7cf51ea"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "ff9bb5a2c40e03b8e8e564081fd0949d"
server_key_iv: "7ad760a82e0d043bb4db5e0ca9d4b33f"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.710493 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/wal/instance:
uuid: "04208839d9c74e7eb1b9db21c7cf51ea"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "ff9bb5a2c40e03b8e8e564081fd0949d"
server_key_iv: "7ad760a82e0d043bb4db5e0ca9d4b33f"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.711688 6948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:54.712407 6963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.712541 6948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:54.712620 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/wal
uuid: "04208839d9c74e7eb1b9db21c7cf51ea"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "ff9bb5a2c40e03b8e8e564081fd0949d"
server_key_iv: "7ad760a82e0d043bb4db5e0ca9d4b33f"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.712669 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:54.740218 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:54.742035 6948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:54.742148 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:54.745803 6948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37499
I20260620 09:16:54.745844 7015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37499 every 8 connection(s)
I20260620 09:16:54.746140 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:16:54.747115 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:54.748611 7016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:16:54.753768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 6948
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:54.756934 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.749290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.0.98.62:41791)
0620 09:16:54.749357 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:54.749359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:54.749514 (+ 155us) server_negotiation.cc:407] Connection header received
0620 09:16:54.749997 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:54.749998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:54.750009 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:54.750066 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:54.750649 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.751158 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.752142 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.752358 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.752411 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.754435 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:54.754436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:54.754436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:54.754443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:54.755714 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.756053 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.756054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.756054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.756108 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.756319 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.756320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.756321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.756463 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:54.756521 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:54.756772 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 09:16:54.756848 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:16:54.757202 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.749231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:57870)
0620 09:16:54.749404 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:54.749441 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:54.749903 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:54.750133 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:54.750137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:54.750381 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:54.750567 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.750571 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.751235 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.751236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.752069 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.752071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.752422 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.752423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.752535 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.752962 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:54.752969 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:54.754352 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:54.755787 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.755792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.755795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.755992 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.756157 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.756158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.756158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.756267 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.756519 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:54.756524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:54.756657 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 09:16:54.756746 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 09:16:54.756882 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260620 09:16:54.757799 7016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:16:54.761495 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.754165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56634 (local address 127.0.98.61:37499)
0620 09:16:54.754294 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:54.754298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:54.754322 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:16:54.754343 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:54.754345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:54.754358 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:54.754423 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:54.754921 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.755595 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.756277 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.756564 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.756626 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.758931 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:54.758938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:54.758938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:54.758945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:54.760469 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.760794 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.760797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.760798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.760851 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.761005 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.761006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.761006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.761136 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:54.761301 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:54.761341 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 09:16:54.761403 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:16:54.764642 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.758031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37499 (local address 127.0.0.1:56636)
0620 09:16:54.758103 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:54.758113 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:54.758158 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:54.758356 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:54.758356 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:54.758500 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:54.758543 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.758544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.759362 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.759363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.760030 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.760032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.760469 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.760469 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.760517 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.760827 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:54.760831 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:54.761914 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:54.763798 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.763799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.763799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.763943 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.764107 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.764108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.764108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.764157 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.764346 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:54.764347 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:54.764381 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:16:54.764439 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 09:16:54.764527 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 09:16:54.764722 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.758109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.0.98.61:37499)
0620 09:16:54.758247 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:54.758249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:54.758255 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:16:54.758271 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:54.758272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:54.758280 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:54.758347 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:54.758620 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.759080 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.760102 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.760402 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.760472 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.761982 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:54.761983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:54.761983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:54.761988 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:54.763737 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.763997 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.763998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.763999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.764054 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.764203 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.764204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.764204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.764298 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:54.764349 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:54.764463 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 09:16:54.764574 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260620 09:16:54.765792 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: Bootstrap starting.
I20260620 09:16:54.766327 7016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:54.766531 7016 log.cc:824] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:54.767143 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: No bootstrap required, opened a new log
I20260620 09:16:54.768524 7016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } }
I20260620 09:16:54.768653 7016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:54.768674 7016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04208839d9c74e7eb1b9db21c7cf51ea, State: Initialized, Role: FOLLOWER
I20260620 09:16:54.768759 7016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } }
I20260620 09:16:54.769027 7016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:54.768985 7023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } } }
I20260620 09:16:54.769081 7023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: This master's current role is: FOLLOWER
W20260620 09:16:54.770444 7034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:16:54.770498 7034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:16:54.772003 6948 master_runner.cc:428] Detected that this master 04208839d9c74e7eb1b9db21c7cf51ea is joining an existing cluster
I20260620 09:16:54.772040 6948 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:37499
I20260620 09:16:54.779966 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.773028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37499 (local address 127.0.0.1:56640)
0620 09:16:54.773182 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:54.773189 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:54.773244 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:54.773418 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:54.773419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:54.773654 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:54.773742 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.773744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.774623 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.774624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.775479 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.775481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.775884 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.775885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.775943 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.776388 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:54.776393 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:54.776798 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:54.778837 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.778838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.778839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.779028 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.779226 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.779227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.779227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.779298 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.779575 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:54.779576 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:54.779645 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 09:16:54.779711 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:16:54.779811 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260620 09:16:54.779979 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.773115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.0.98.61:37499)
0620 09:16:54.773147 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:54.773149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:54.773247 (+ 98us) server_negotiation.cc:407] Connection header received
0620 09:16:54.773298 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:54.773299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:54.773310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:54.773385 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:54.773803 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.774539 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.775556 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.775805 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.775866 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.776879 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:54.776880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:54.776881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:54.776888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:54.778756 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.779102 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.779103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.779103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.779162 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.779354 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.779355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.779355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.779491 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:54.779552 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:54.779736 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 09:16:54.779872 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":17}
I20260620 09:16:54.780274 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.773019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.0.98.62:41791)
0620 09:16:54.773061 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:54.773063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:54.773235 (+ 172us) server_negotiation.cc:407] Connection header received
0620 09:16:54.773335 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:54.773336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:54.773346 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:54.773397 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:54.773771 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.774530 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.776549 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.776835 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.776893 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.777816 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:54.777817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:54.777817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:54.777824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:54.779176 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.779490 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.779491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.779491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.779542 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.779756 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.779757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.779757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.779878 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:54.779918 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:54.780116 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 09:16:54.780185 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:16:54.780649 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.772969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:57880)
0620 09:16:54.773159 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:54.773180 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:54.773257 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:54.773402 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:54.773403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:54.773616 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:54.773704 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.773706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.775589 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.775589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.776459 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.776462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.776942 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.776942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.776987 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.777368 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:54.777373 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:54.777742 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:54.779242 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.779243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.779243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.779435 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.779591 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.779592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.779592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.779685 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.779960 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:54.779961 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:54.780010 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 09:16:54.780368 (+ 358us) client_negotiation.cc:240] Negotiation successful
0620 09:16:54.780569 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":13,"thread_start_us":86,"threads_started":1}
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:16:54.789594 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.782571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.0.98.61:37499)
0620 09:16:54.782604 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:54.782606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:54.782763 (+ 157us) server_negotiation.cc:407] Connection header received
0620 09:16:54.782828 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:54.782829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:54.782839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:54.782901 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:54.783252 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.783999 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.784714 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.785005 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.785078 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.787208 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:54.787209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:54.787209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:54.787215 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:54.788420 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.788899 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.788900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.788900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.788957 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.789138 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.789138 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.789138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.789251 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:54.789296 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:54.789469 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 09:16:54.789528 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 09:16:54.789705 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.782532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37499 (local address 127.0.0.1:56642)
0620 09:16:54.782681 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:54.782714 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:54.782773 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:54.782903 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:54.782905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:54.783139 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:54.783193 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.783195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.784071 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.784071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.784647 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.784649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.785064 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.785065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.785113 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.785547 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:54.785551 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:54.787126 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:54.788492 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.788500 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.788503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.788812 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.789021 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.789022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.789023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.789087 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.789302 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:54.789309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:54.789366 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 09:16:54.789452 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:16:54.789572 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260620 09:16:54.790241 6484 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:37499: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 37499
}
attrs {
promote: true
}
}
dest_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca"
cas_config_opid_index: -1
I20260620 09:16:54.790529 6484 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true } }
I20260620 09:16:54.790949 7044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 04208839d9c74e7eb1b9db21c7cf51ea (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true } } }
I20260620 09:16:54.791303 6990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:54.792013 7044 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:37499
I20260620 09:16:54.792184 6990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 FOLLOWER]: Refusing update from remote peer a4bd2c7b810f4490a53b96a66bd7a3ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 09:16:54.792232 6948 master.cc:537] Master@127.0.98.61:37499 shutting down...
I20260620 09:16:54.792227 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true } } }
I20260620 09:16:54.792285 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:54.792567 7043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:16:54.793180 6470 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 09:16:54.793529 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca -> Peer 04208839d9c74e7eb1b9db21c7cf51ea (127.0.98.61:37499): Couldn't send request to peer 04208839d9c74e7eb1b9db21c7cf51ea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 09:16:54.794821 6948 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:54.794893 6948 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:54.794939 6948 tablet_replica.cc:330] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: stopping tablet replica
I20260620 09:16:54.807405 6948 master.cc:552] Master@127.0.98.61:37499 shutdown complete.
I20260620 09:16:54.807468 6948 master_runner.cc:305] Clearing existing system tablet
I20260620 09:16:54.807639 6948 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:54.807693 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:54.808311 6948 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:54.808954 7048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.809067 6948 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:54.809122 6948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:16:54.809147 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/wal
uuid: "04208839d9c74e7eb1b9db21c7cf51ea"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "ff9bb5a2c40e03b8e8e564081fd0949d"
server_key_iv: "7ad760a82e0d043bb4db5e0ca9d4b33f"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.809803 6948 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:54.811048 6948 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 09:16:54.811110 6948 log.cc:1196] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 09:16:54.811304 6948 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: Deleting consensus metadata
I20260620 09:16:54.811408 6948 master_runner.cc:315] Copying system tablet from 127.0.98.62:41791
I20260620 09:16:54.812114 6948 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41791
I20260620 09:16:54.818545 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.812688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.0.98.62:41791)
0620 09:16:54.812735 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:54.812737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:54.812746 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:16:54.812772 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:54.812773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:54.812783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:54.812847 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:54.813233 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.814011 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.814742 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.815044 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.815115 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.816074 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:54.816076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:54.816076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:54.816084 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:54.817552 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.817844 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.817844 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.817845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.817904 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.818076 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.818077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.818077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.818193 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:54.818250 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:54.818382 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 09:16:54.818451 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:16:54.818557 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.812443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:57884)
0620 09:16:54.812579 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:54.812591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:54.812654 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:54.812884 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:54.812885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:54.813038 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:54.813099 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.813101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.814100 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.814101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.814668 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:54.814670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.815131 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.815132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:54.815200 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.815697 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:54.815702 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:54.815992 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:54.817628 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.817629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.817629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.817785 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.817962 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:54.817962 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:54.817962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:54.818002 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:54.818247 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:54.818248 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:54.818290 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 09:16:54.818352 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 09:16:54.818452 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260620 09:16:54.818853 6504 tablet_copy_service.cc:140] P a4bd2c7b810f4490a53b96a66bd7a3ca: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 04208839d9c74e7eb1b9db21c7cf51ea ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57884)
I20260620 09:16:54.818931 6504 tablet_copy_service.cc:161] P a4bd2c7b810f4490a53b96a66bd7a3ca: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 04208839d9c74e7eb1b9db21c7cf51ea at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57884: session id = 04208839d9c74e7eb1b9db21c7cf51ea-00000000000000000000000000000000
I20260620 09:16:54.819526 6504 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 09:16:54.820180 6948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 09:16:54.821414 6948 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: tablet copy: Starting download of 0 data blocks...
I20260620 09:16:54.821554 6948 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: tablet copy: Starting download of 1 WAL segments...
I20260620 09:16:54.822340 6948 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 09:16:54.823200 6504 tablet_copy_service.cc:342] P a4bd2c7b810f4490a53b96a66bd7a3ca: Request end of tablet copy session 04208839d9c74e7eb1b9db21c7cf51ea-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57884
I20260620 09:16:54.823256 6504 tablet_copy_service.cc:433] P a4bd2c7b810f4490a53b96a66bd7a3ca: ending tablet copy session 04208839d9c74e7eb1b9db21c7cf51ea-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 04208839d9c74e7eb1b9db21c7cf51ea
I20260620 09:16:54.825394 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:54.826462 7055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:54.826475 7056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:54.826653 6948 server_base.cc:1034] running on GCE node
W20260620 09:16:54.826742 7058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:54.826871 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:54.826995 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:54.828100 6948 hybrid_clock.cc:648] HybridClock initialized: now 1781947014828094 us; error 22 us; skew 500 ppm
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:54.829392 6948 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 09:16:54.829618 6948 webserver.cc:460] Webserver started at http://127.0.98.61:41233/ using document root <none> and password file <none>
I20260620 09:16:54.829743 6948 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:54.829771 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:54.830292 6948 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.830746 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.830876 6948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.830920 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/wal
uuid: "04208839d9c74e7eb1b9db21c7cf51ea"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "ff9bb5a2c40e03b8e8e564081fd0949d"
server_key_iv: "7ad760a82e0d043bb4db5e0ca9d4b33f"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.830974 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:54.836881 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:54.838284 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:54.840881 6948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37499
I20260620 09:16:54.841228 7117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37499 every 8 connection(s)
I20260620 09:16:54.841243 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:16:54.841945 7118 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:16:54.842336 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: Bootstrap starting.
I20260620 09:16:54.844400 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:16:54.844578 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: Bootstrap complete.
I20260620 09:16:54.844744 7118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true } }
I20260620 09:16:54.844807 7118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 04208839d9c74e7eb1b9db21c7cf51ea, State: Initialized, Role: LEARNER
I20260620 09:16:54.844830 7118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true } }
I20260620 09:16:54.844990 7118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:54.844993 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true } } }
I20260620 09:16:54.845105 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: This master's current role is: LEARNER
I20260620 09:16:54.846218 7132 catalog_manager.cc:1259] Loaded cluster ID: dbf97b58aadf4ef59ca61a172f58a443
I20260620 09:16:54.846256 7132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: loading cluster ID for follower catalog manager: success
I20260620 09:16:54.847129 7132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: acquiring CA information for follower catalog manager: success
I20260620 09:16:54.847369 7132 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:16:54.869127 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:54.863204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.0.98.61:37499)
0620 09:16:54.863313 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:54.863315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:54.863322 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:54.863342 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:54.863343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:54.863351 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:54.863401 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:54.863957 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.864613 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.865295 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:54.865493 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:54.865546 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:54.866713 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:54.866715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:54.866715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:54.866721 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:54.868157 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.868444 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.868445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.868445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.868492 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:54.868647 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:54.868648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:54.868648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:54.868760 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:54.868843 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:54.868952 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 09:16:54.869021 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 09:16:54.891463 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43625
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41791,127.0.98.61:37499,127.0.98.60:43625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:54.975091 7136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:54.975281 7136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:54.975301 7136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:54.976413 7136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:54.976450 7136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:54.976464 7136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:54.976475 7136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:54.976485 7136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:54.977892 7136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41791,127.0.98.61:37499,127.0.98.60:43625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:54.978075 7136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:54.978233 7136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:54.982597 7142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:54.982597 7141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:54.982594 7144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:54.982735 7136 server_base.cc:1034] running on GCE node
I20260620 09:16:54.983057 7136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:54.983336 7136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:54.984479 7136 hybrid_clock.cc:648] HybridClock initialized: now 1781947014984467 us; error 30 us; skew 500 ppm
Jun 20 09:16:54 dist-test-slave-rq8c krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:54.986789 7136 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 09:16:54.987175 7136 webserver.cc:460] Webserver started at http://127.0.98.60:37457/ using document root <none> and password file <none>
I20260620 09:16:54.987360 7136 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:54.987401 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:54.987515 7136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:54.988552 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data/instance:
uuid: "83aae58fc5f546a9a9b928944595c5c5"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "3845d194750a32ae1cdc4f3bf5352853"
server_key_iv: "32652947e58773d78d25a573855a1398"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.988864 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/wal/instance:
uuid: "83aae58fc5f546a9a9b928944595c5c5"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "3845d194750a32ae1cdc4f3bf5352853"
server_key_iv: "32652947e58773d78d25a573855a1398"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.990000 7136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:16:54.990588 7151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:54.990718 7136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:54.990795 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/wal
uuid: "83aae58fc5f546a9a9b928944595c5c5"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "3845d194750a32ae1cdc4f3bf5352853"
server_key_iv: "32652947e58773d78d25a573855a1398"
server_key_version: "encryptionkey@0"
I20260620 09:16:54.990851 7136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:55.004733 7136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:55.006554 7136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:55.006690 7136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:55.010260 7136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43625
I20260620 09:16:55.010293 7203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43625 every 8 connection(s)
I20260620 09:16:55.010609 7136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:16:55.011497 7204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:55.012974 7204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:16:55.016034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 7136
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:16:55.021653 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.013768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.0.98.62:41791)
0620 09:16:55.013813 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.013815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.013897 (+ 82us) server_negotiation.cc:407] Connection header received
0620 09:16:55.014341 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.014343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.014353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.014407 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.014958 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.015441 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.016631 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.016895 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.016946 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.019084 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.019085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.019085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.019092 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.020420 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.020788 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.020789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.020789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.020849 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.021058 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.021059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.021059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.021171 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.021229 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.021465 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.021553 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:16:55.021773 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.013636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:57896)
0620 09:16:55.013784 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.013823 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.014261 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.014467 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.014470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.014717 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.014897 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.014902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.015497 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.015498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.016568 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.016570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.016967 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.016968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.017112 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.017658 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.017667 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.019014 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.020493 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.020499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.020501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.020713 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.020909 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.020910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.020910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.020995 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.021227 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.021233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.021350 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.021436 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.021573 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260620 09:16:55.022262 7204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:16:55.024255 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.016454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.0.98.60:43625)
0620 09:16:55.016588 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.016592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.016610 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:16:55.016680 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.016683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.016703 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.016773 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.017387 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.018178 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.018901 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.019262 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.019317 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.021395 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.021401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.021401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.021408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.023292 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.023577 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.023579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.023580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.023666 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.023823 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.023823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.023824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.023937 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.024043 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.024131 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.024187 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:16:55.028069 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.022473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37499 (local address 127.0.0.1:56670)
0620 09:16:55.022504 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.022514 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.022549 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.022736 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.022737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.022830 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.022856 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.022857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.023670 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.023671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.024360 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.024363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.024736 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.024737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.024792 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.025075 (+ 283us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.025079 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.026034 (+ 955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.027380 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.027381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.027381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.027502 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.027669 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.027670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.027670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.027717 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.027884 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.027885 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.027915 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.027957 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.028014 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 09:16:55.028095 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.022566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.0.98.61:37499)
0620 09:16:55.022621 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.022623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.022631 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:55.022658 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.022660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.022672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.022732 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.022920 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.023568 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.024430 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.024673 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.024731 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.026108 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.026109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.026109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.026115 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.027325 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.027540 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.027541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.027541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.027590 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.027757 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.027758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.027758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.027846 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.027887 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.027973 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.028028 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:16:55.028543 7204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:16:55.033789 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.028743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43625 (local address 127.0.0.1:53316)
0620 09:16:55.028778 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.028787 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.028815 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.028911 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.028912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.029003 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.029027 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.029027 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.029568 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.029569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.030027 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.030028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.030293 (+ 265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.030293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.030324 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.030662 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.030666 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.031786 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.033119 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.033119 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.033120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.033264 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.033404 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.033405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.033405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.033446 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.033589 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.033590 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.033618 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.033659 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.033721 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 09:16:55.033841 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.028797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.0.98.60:43625)
0620 09:16:55.028826 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.028827 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.028834 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:55.028850 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.028851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.028860 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.028907 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.029067 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.029520 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.030075 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.030251 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.030291 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.031859 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.031860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.031860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.031866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.033050 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.033315 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.033316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.033316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.033356 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.033481 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.033481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.033481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.033558 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.033595 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.033681 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.033760 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":14}
I20260620 09:16:55.035010 7204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: Bootstrap starting.
I20260620 09:16:55.036213 7204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:55.036422 7204 log.cc:824] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:55.037011 7204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: No bootstrap required, opened a new log
I20260620 09:16:55.038300 7204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } }
I20260620 09:16:55.038422 7204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:55.038442 7204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83aae58fc5f546a9a9b928944595c5c5, State: Initialized, Role: FOLLOWER
I20260620 09:16:55.038528 7204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } }
I20260620 09:16:55.038779 7204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:55.038749 7209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } } }
I20260620 09:16:55.038831 7209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 09:16:55.040292 7220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:16:55.040342 7220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:16:55.042099 7136 master_runner.cc:428] Detected that this master 83aae58fc5f546a9a9b928944595c5c5 is joining an existing cluster
I20260620 09:16:55.042138 7136 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:43625
I20260620 09:16:55.048584 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.043308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.0.98.62:41791)
0620 09:16:55.043352 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.043354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.043360 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:16:55.043379 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.043380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.043389 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.043433 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.043769 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.044252 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.045043 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.045305 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.045357 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.046170 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.046172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.046173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.046179 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.047381 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.047694 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.047695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.047695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.047749 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.047939 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.047941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.047941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.048090 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.048177 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.048407 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.048493 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:16:55.048961 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.042954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:57906)
0620 09:16:55.043093 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.043103 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.043150 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.043445 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.043446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.043597 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.043700 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.043702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.044315 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.044316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.044975 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.044977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.045361 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.045362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.045421 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.045844 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.045849 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.046102 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.047460 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.047461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.047462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.047608 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.047799 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.047800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.047800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.047856 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.048205 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.048206 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.048250 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.048746 (+ 496us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.048853 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":2,"thread_start_us":98,"threads_started":1}
I20260620 09:16:55.054517 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.043107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.0.98.61:37499)
0620 09:16:55.043145 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.043147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.047906 (+ 4759us) server_negotiation.cc:407] Connection header received
0620 09:16:55.047966 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.047967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.047979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.048024 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.048465 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.049219 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.050740 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.051011 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.051046 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.051941 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.051942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.051943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.051966 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.053475 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.053780 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.053781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.053781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.053833 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.054003 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.054004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.054004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.054122 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.054174 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.054362 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.054427 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:16:55.054531 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.042964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37499 (local address 127.0.0.1:56672)
0620 09:16:55.047856 (+ 4892us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.047866 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.047915 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.048073 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.048074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.048323 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.048382 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.048384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.049834 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.049835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.050661 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.050662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.051117 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.051117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.051196 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.051531 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.051535 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.051874 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.053562 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.053563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.053563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.053719 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.053884 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.053884 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.053884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.053937 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.054218 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.054218 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.054260 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.054334 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.054427 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4866,"thread_start_us":76,"threads_started":1}
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947014, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:16:55.058269 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.051333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53332 (local address 127.0.98.60:43625)
0620 09:16:55.051548 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.051550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.051557 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:55.051590 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.051591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.051600 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.051691 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.051976 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.052641 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.053571 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.053888 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.053961 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.056087 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.056088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.056088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.056094 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.057286 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.057621 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.057622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.057622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.057661 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.057819 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.057819 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.057819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.057916 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.057958 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.058129 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.058208 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
I20260620 09:16:55.058286 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.051258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43625 (local address 127.0.0.1:53332)
0620 09:16:55.051335 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.051345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.051377 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.051702 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.051703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.051862 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.051905 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.051907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.052737 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.052738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.053498 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.053500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.053964 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.053965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.054033 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.054416 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.054421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.056017 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.057354 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.057355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.057355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.057560 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.057714 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.057715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.057715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.057772 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.057966 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.057966 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.058011 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.058080 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.058186 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 09:16:55.058705 6484 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:43625: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 43625
}
attrs {
promote: true
}
}
dest_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca"
cas_config_opid_index: 5
I20260620 09:16:55.058923 6484 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } }
I20260620 09:16:55.059367 7044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 83aae58fc5f546a9a9b928944595c5c5 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.059463 7178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:55.060104 7044 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:43625
I20260620 09:16:55.060178 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.060236 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:55.060392 7136 master.cc:537] Master@127.0.98.60:43625 shutting down...
I20260620 09:16:55.060803 7178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 1 FOLLOWER]: Refusing update from remote peer a4bd2c7b810f4490a53b96a66bd7a3ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 09:16:55.061116 7043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:16:55.061772 6468 consensus_peers.cc:487] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca -> Peer 83aae58fc5f546a9a9b928944595c5c5 (127.0.98.60:43625): Couldn't send request to peer 83aae58fc5f546a9a9b928944595c5c5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 09:16:55.063098 7136 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:55.063184 7136 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:55.063231 7136 tablet_replica.cc:330] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: stopping tablet replica
I20260620 09:16:55.064127 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.059287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37499 (local address 127.0.0.1:56680)
0620 09:16:55.061006 (+ 1719us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.061017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.061067 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.061287 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.061288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.061290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:55.061328 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.061329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.062133 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.062134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.063196 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.063199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.063896 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.063897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.063954 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.063971 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.064025 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.064068 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1702}
I20260620 09:16:55.064127 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.059368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.0.98.61:37499)
0620 09:16:55.059400 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.059401 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.061114 (+ 1713us) server_negotiation.cc:407] Connection header received
0620 09:16:55.061146 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.061147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.061170 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.061220 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:55.061394 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.062056 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.063292 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.063826 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.063888 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.063962 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.064030 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.064063 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:16:55.064385 7092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 LEARNER]: Refusing update from remote peer a4bd2c7b810f4490a53b96a66bd7a3ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 09:16:55.064566 7043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 09:16:55.065102 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 83aae58fc5f546a9a9b928944595c5c5 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.066152 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.066234 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: This master's current role is: LEARNER
I20260620 09:16:55.066326 7043 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca: attempting to promote NON_VOTER 04208839d9c74e7eb1b9db21c7cf51ea to VOTER
I20260620 09:16:55.066358 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4bd2c7b810f4490a53b96a66bd7a3ca. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: true } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.066419 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: This master's current role is: LEARNER
I20260620 09:16:55.066501 7043 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } }
I20260620 09:16:55.067008 7092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 LEARNER]: Refusing update from remote peer a4bd2c7b810f4490a53b96a66bd7a3ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260620 09:16:55.067121 6468 consensus_peers.cc:487] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca -> Peer 83aae58fc5f546a9a9b928944595c5c5 (127.0.98.60:43625): Couldn't send request to peer 83aae58fc5f546a9a9b928944595c5c5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 09:16:55.067196 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 09:16:55.067704 7232 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 04208839d9c74e7eb1b9db21c7cf51ea (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.067842 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 04208839d9c74e7eb1b9db21c7cf51ea (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.068416 7232 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.068487 7232 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:55.068480 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.068627 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:55.068655 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.068714 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:16:55.079064 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.043173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43625 (local address 127.0.0.1:53318)
0620 09:16:55.047996 (+ 4823us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.048004 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.048053 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.048179 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.048180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.048343 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.048673 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.048675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.049461 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.049461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.050194 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.050196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.050658 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.050659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.050710 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.076137 (+ 25427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.076142 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.076535 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.078179 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.078179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.078180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.078386 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.078554 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.078554 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.078555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.078597 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.078809 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.078810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.078859 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.078922 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.078998 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4812,"thread_start_us":55,"threads_started":1}
I20260620 09:16:55.079191 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.043202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.0.98.60:43625)
0620 09:16:55.043230 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.043231 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.048038 (+ 4807us) server_negotiation.cc:407] Connection header received
0620 09:16:55.048103 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.048104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.048111 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.048135 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.048746 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.049393 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.050269 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.050544 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.050584 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.076618 (+ 26034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.076619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.076619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.076626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.078102 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.078459 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.078460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.078460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.078503 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.078646 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.078646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.078646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.078760 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.078818 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.079013 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.079089 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":15}
W20260620 09:16:55.080194 6437 connection.cc:535] client connection to 127.0.98.60:43625 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:16:55.082758 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.082643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43625 (local address 127.0.0.1:53348)
0620 09:16:55.082711 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.082738 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 09:16:55.091487 7136 master.cc:552] Master@127.0.98.60:43625 shutdown complete.
I20260620 09:16:55.091536 7136 master_runner.cc:305] Clearing existing system tablet
I20260620 09:16:55.091682 7136 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:55.091728 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:55.092433 7136 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:55.094194 7234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.094301 7136 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:55.094347 7136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260620 09:16:55.094367 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/wal
uuid: "83aae58fc5f546a9a9b928944595c5c5"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "3845d194750a32ae1cdc4f3bf5352853"
server_key_iv: "32652947e58773d78d25a573855a1398"
server_key_version: "encryptionkey@0"
I20260620 09:16:55.095214 7136 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:55.096604 7136 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 09:16:55.096671 7136 log.cc:1196] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 09:16:55.096848 7136 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: Deleting consensus metadata
I20260620 09:16:55.096954 7136 master_runner.cc:315] Copying system tablet from 127.0.98.62:41791
I20260620 09:16:55.097565 7136 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41791
I20260620 09:16:55.103480 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.097918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.0.98.62:41791)
0620 09:16:55.097963 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.097965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.098095 (+ 130us) server_negotiation.cc:407] Connection header received
0620 09:16:55.098168 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.098169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.098178 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.098222 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.098743 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.099210 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.100226 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.100399 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.100444 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.101375 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.101376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.101376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.101383 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.102592 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.102934 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.102935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.102935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.102979 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.103111 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.103112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.103112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.103207 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.103249 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.103370 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.103423 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:16:55.103538 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.097866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:57912)
0620 09:16:55.098018 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.098028 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.098099 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.098228 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.098229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.098421 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.098675 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.098677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.099273 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.099274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.100157 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.100160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.100456 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.100457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.100523 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.100911 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.100916 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.101288 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.102665 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.102666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.102666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.102873 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.103014 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.103014 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.103014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.103065 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.103251 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.103252 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.103295 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.103349 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.103442 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260620 09:16:55.103902 6504 tablet_copy_service.cc:140] P a4bd2c7b810f4490a53b96a66bd7a3ca: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 83aae58fc5f546a9a9b928944595c5c5 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57912)
I20260620 09:16:55.103960 6504 tablet_copy_service.cc:161] P a4bd2c7b810f4490a53b96a66bd7a3ca: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 83aae58fc5f546a9a9b928944595c5c5 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57912: session id = 83aae58fc5f546a9a9b928944595c5c5-00000000000000000000000000000000
I20260620 09:16:55.104508 6504 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 09:16:55.104941 7136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 09:16:55.105760 7136 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: tablet copy: Starting download of 0 data blocks...
I20260620 09:16:55.105885 7136 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: tablet copy: Starting download of 1 WAL segments...
I20260620 09:16:55.106554 7136 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 09:16:55.107239 6504 tablet_copy_service.cc:342] P a4bd2c7b810f4490a53b96a66bd7a3ca: Request end of tablet copy session 83aae58fc5f546a9a9b928944595c5c5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57912
I20260620 09:16:55.107281 6504 tablet_copy_service.cc:433] P a4bd2c7b810f4490a53b96a66bd7a3ca: ending tablet copy session 83aae58fc5f546a9a9b928944595c5c5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 83aae58fc5f546a9a9b928944595c5c5
I20260620 09:16:55.109727 7136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:55.110831 7241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:55.111003 7244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:55.111132 7242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:55.111217 7136 server_base.cc:1034] running on GCE node
I20260620 09:16:55.111301 7136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:55.111418 7136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:55.112532 7136 hybrid_clock.cc:648] HybridClock initialized: now 1781947015112519 us; error 28 us; skew 500 ppm
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:55.113829 7136 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 09:16:55.114084 7136 webserver.cc:460] Webserver started at http://127.0.98.60:41889/ using document root <none> and password file <none>
I20260620 09:16:55.114202 7136 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:55.114236 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:55.114846 7136 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:55.115305 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.115413 7136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.115446 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/wal
uuid: "83aae58fc5f546a9a9b928944595c5c5"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "3845d194750a32ae1cdc4f3bf5352853"
server_key_iv: "32652947e58773d78d25a573855a1398"
server_key_version: "encryptionkey@0"
I20260620 09:16:55.115496 7136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:55.123835 7136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:55.125185 7136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:55.128232 7136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43625
I20260620 09:16:55.128351 7303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43625 every 8 connection(s)
I20260620 09:16:55.128528 7136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:16:55.129170 7304 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:16:55.129549 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: Bootstrap starting.
I20260620 09:16:55.132296 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:16:55.132560 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: Bootstrap complete.
I20260620 09:16:55.132766 7304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } }
I20260620 09:16:55.132858 7304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83aae58fc5f546a9a9b928944595c5c5, State: Initialized, Role: LEARNER
I20260620 09:16:55.132901 7304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } }
I20260620 09:16:55.133070 7304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:55.133064 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.133286 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: This master's current role is: LEARNER
I20260620 09:16:55.134809 7319 catalog_manager.cc:1259] Loaded cluster ID: dbf97b58aadf4ef59ca61a172f58a443
I20260620 09:16:55.134848 7319 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: loading cluster ID for follower catalog manager: success
I20260620 09:16:55.136222 7319 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: acquiring CA information for follower catalog manager: success
I20260620 09:16:55.136516 7319 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:16:55.137368 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.130692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.0.98.60:43625)
0620 09:16:55.130807 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.130809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.130905 (+ 96us) server_negotiation.cc:407] Connection header received
0620 09:16:55.130923 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.130923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.130931 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.130980 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.131484 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.132287 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.133046 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.133331 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.133419 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.134680 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.134681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.134681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.134688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.136339 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.136698 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.136699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.136699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.136755 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.136910 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.136911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.136912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.137009 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.137098 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.137181 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.137272 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
I20260620 09:16:55.137821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 6546
I20260620 09:16:55.142607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46105
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41791
--webserver_port=35061
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41791,127.0.98.61:37499
--tserver_master_addrs=127.0.98.62:41791,127.0.98.61:37499,127.0.98.60:43625 with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:55.220294 7320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:55.220479 7320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:55.220501 7320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:55.221582 7320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:55.221622 7320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:55.221653 7320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:55.223037 7320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35061
--tserver_master_addrs=127.0.98.62:41791,127.0.98.61:37499,127.0.98.60:43625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:55.223238 7320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:55.223455 7320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:55.228358 7325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:55.228434 7320 server_base.cc:1034] running on GCE node
W20260620 09:16:55.228399 7328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:55.228389 7326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:55.228688 7320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:55.228894 7320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:55.230027 7320 hybrid_clock.cc:648] HybridClock initialized: now 1781947015230014 us; error 26 us; skew 500 ppm
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:55.232237 7320 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 09:16:55.232609 7320 webserver.cc:460] Webserver started at http://127.0.98.1:35061/ using document root <none> and password file <none>
I20260620 09:16:55.232841 7320 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:55.232901 7320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:55.234161 7320 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:55.234611 7335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.234776 7320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.234854 7320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "8587525e365e445c8df92975c7dbd21c"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "255ded57396df5fa70b04f808ae9ed8e"
server_key_iv: "3072f3b08b57126d2879819233476cc1"
server_key_version: "encryptionkey@0"
I20260620 09:16:55.235145 7320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:55.245527 7320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:55.247239 7320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:55.247337 7320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:55.247495 7320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:55.247761 7320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:55.247793 7320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.247812 7320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:55.247828 7320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.253317 7320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46105
I20260620 09:16:55.253389 7448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46105 every 8 connection(s)
I20260620 09:16:55.253667 7320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:16:55.257071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 7320
I20260620 09:16:55.257218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 6680
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:55.263700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40687
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41791
--webserver_port=44057
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41791,127.0.98.61:37499
--tserver_master_addrs=127.0.98.62:41791,127.0.98.61:37499,127.0.98.60:43625 with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:16:55.264757 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.254348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60253 (local address 127.0.98.62:41791)
0620 09:16:55.254388 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.254391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.254812 (+ 421us) server_negotiation.cc:407] Connection header received
0620 09:16:55.255190 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.255192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.255206 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.255263 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.257222 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.257953 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.258908 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.259160 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.259192 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.262017 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.262019 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.262019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.262026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.263357 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.263776 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.263777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.263778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.263833 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.264116 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.264117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.264118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.264230 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.264275 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.264596 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.264671 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:16:55.264829 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.254696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54725 (local address 127.0.98.60:43625)
0620 09:16:55.254739 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.254740 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.255292 (+ 552us) server_negotiation.cc:407] Connection header received
0620 09:16:55.255383 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.255384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.255400 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.255456 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.256072 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.256575 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.258269 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.258600 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.258639 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.261658 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.261660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.261660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.261667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.263193 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.263988 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.263989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.263989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.264040 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.264247 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.264248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.264248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.264368 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.264426 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.264638 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.264736 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:16:55.264921 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.254324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.1:60253)
0620 09:16:55.254737 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.254748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.255094 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.255336 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.255340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.255794 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.255972 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.255976 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.258064 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.258065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.258824 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.258826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.259249 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.259250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.259294 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.260588 (+ 1294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.260593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.261946 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.263416 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.263417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.263418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.263709 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.263896 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.263897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.263897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.264035 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.264306 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.264312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.264461 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.264549 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.264713 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":3,"spinlock_wait_cycles":7040,"thread_start_us":108,"threads_started":1}
I20260620 09:16:55.265355 7450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41791
I20260620 09:16:55.265450 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.254624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43625 (local address 127.0.98.1:54725)
0620 09:16:55.255238 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.255247 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.255313 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.255473 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.255474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.255752 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.255972 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.255977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.256661 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.256662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.258180 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.258182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.258735 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.258736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.258866 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.259999 (+ 1133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.260009 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.261544 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.263277 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.263283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.263285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.263705 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.264103 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.264104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.264105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.264176 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.264426 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.264426 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.264472 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.265214 (+ 742us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.265315 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":8,"thread_start_us":61,"threads_started":1}
I20260620 09:16:55.265470 7450 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:55.265656 7450 heartbeater.cc:511] Master 127.0.98.62:41791 requested a full tablet report, sending...
I20260620 09:16:55.265785 7449 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43625
I20260620 09:16:55.265826 7449 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:55.265883 7449 heartbeater.cc:511] Master 127.0.98.60:43625 requested a full tablet report, sending...
I20260620 09:16:55.266040 6484 ts_manager.cc:194] Re-registered known tserver with Master: 8587525e365e445c8df92975c7dbd21c (127.0.98.1:46105)
I20260620 09:16:55.266382 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60253
I20260620 09:16:55.266888 7268 ts_manager.cc:194] Registered new tserver with Master: 8587525e365e445c8df92975c7dbd21c (127.0.98.1:46105)
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:16:55.270812 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.254300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35645 (local address 127.0.98.61:37499)
0620 09:16:55.254391 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.254392 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.254612 (+ 220us) server_negotiation.cc:407] Connection header received
0620 09:16:55.255184 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.255186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.255198 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.255243 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.256664 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.257120 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.266509 (+ 9389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.266797 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.266858 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.268650 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.268651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.268651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.268658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.269838 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.270197 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.270197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.270198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.270236 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.270406 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.270407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.270407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.270496 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.270535 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.270703 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.270755 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 09:16:55.270908 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.254324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37499 (local address 127.0.98.1:35645)
0620 09:16:55.254498 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.254530 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.255084 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.255335 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.255340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.255768 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.255972 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.255977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.264922 (+ 8945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.264924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.266380 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.266382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.266907 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.266908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.267028 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.267420 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.267424 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.268562 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.269913 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.269915 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.269915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.270127 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.270287 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.270288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.270288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.270352 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.270548 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.270549 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.270606 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.270682 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.270789 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":5,"thread_start_us":107,"threads_started":1}
I20260620 09:16:55.271272 7451 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37499
I20260620 09:16:55.271320 7451 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:55.271371 7451 heartbeater.cc:511] Master 127.0.98.61:37499 requested a full tablet report, sending...
I20260620 09:16:55.271744 7082 ts_manager.cc:194] Registered new tserver with Master: 8587525e365e445c8df92975c7dbd21c (127.0.98.1:46105)
W20260620 09:16:55.344892 7456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:55.345080 7456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:55.345109 7456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:55.346258 7456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:55.346297 7456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:55.346333 7456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:55.347808 7456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44057
--tserver_master_addrs=127.0.98.62:41791,127.0.98.61:37499,127.0.98.60:43625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:55.348027 7456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:55.348217 7456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:16:55.352996 7456 server_base.cc:1034] running on GCE node
W20260620 09:16:55.352913 7461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:55.352936 7464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:55.352914 7462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:55.353287 7456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:55.353509 7456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:55.354656 7456 hybrid_clock.cc:648] HybridClock initialized: now 1781947015354643 us; error 29 us; skew 500 ppm
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:55.356900 7456 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 09:16:55.357280 7456 webserver.cc:460] Webserver started at http://127.0.98.2:44057/ using document root <none> and password file <none>
I20260620 09:16:55.357466 7456 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:55.357522 7456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:55.358824 7456 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:55.359328 7471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.359474 7456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.359551 7456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "fbc5044d7eec4e3db65214b485134ca8"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "ada662b47bb91121c245589b754eea69"
server_key_iv: "97d89e14fc464d6abfef5fbad89f9a83"
server_key_version: "encryptionkey@0"
I20260620 09:16:55.359959 7456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:55.372767 7456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:55.374475 7456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:55.374594 7456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:55.374787 7456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:55.375052 7456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:55.375083 7456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.375113 7456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:55.375130 7456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.381209 7456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40687
I20260620 09:16:55.381273 7584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40687 every 8 connection(s)
I20260620 09:16:55.381561 7456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:16:55.389016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 7456
I20260620 09:16:55.389171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 6814
I20260620 09:16:55.391546 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.382237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44625 (local address 127.0.98.62:41791)
0620 09:16:55.382281 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.382283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.382666 (+ 383us) server_negotiation.cc:407] Connection header received
0620 09:16:55.383053 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.383055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.383069 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.383144 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.384555 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.385013 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.385858 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.386056 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.386109 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.388260 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.388261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.388261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.388268 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.390089 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.390537 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.390538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.390539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.390595 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.390843 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.390844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.390844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.390969 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.391006 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.391374 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.391454 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:16:55.391922 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.382586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54945 (local address 127.0.98.60:43625)
0620 09:16:55.382626 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.382627 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.383150 (+ 523us) server_negotiation.cc:407] Connection header received
0620 09:16:55.383230 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.383231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.383243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.383303 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.383973 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.384466 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.385650 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.385935 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.386017 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.388693 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.388694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.388694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.388701 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.390777 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.391155 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.391156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.391156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.391208 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.391391 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.391392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.391392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.391512 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.391551 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.391764 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.391832 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:16:55.392400 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.382488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43625 (local address 127.0.98.2:54945)
0620 09:16:55.383091 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.383099 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.383175 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.383328 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.383329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.383660 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.383875 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.383881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.384549 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.384549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.385548 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.385550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.386085 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.386086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.386225 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.386776 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.386781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.388608 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.390860 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.390861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.390861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.391072 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.391275 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.391275 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.391276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.391328 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.391594 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.391595 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.391656 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.392006 (+ 350us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.392166 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":6,"thread_start_us":59,"threads_started":1}
I20260620 09:16:55.392401 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.382227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.2:44625)
0620 09:16:55.382601 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.382610 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.382941 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.383220 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.383224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.383695 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.383875 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.383881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.385087 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.385088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.385796 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.385797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.386112 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.386112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.386223 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.386661 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.386671 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.388178 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.390177 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.390184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.390186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.390452 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.390661 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.390661 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.390662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.390779 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.391048 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.391053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.391228 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.391895 (+ 667us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.392223 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":36,"spinlock_wait_cycles":384,"thread_start_us":82,"threads_started":1}
I20260620 09:16:55.392822 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.382204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45817 (local address 127.0.98.61:37499)
0620 09:16:55.382281 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.382283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.382497 (+ 214us) server_negotiation.cc:407] Connection header received
0620 09:16:55.383053 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.383055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.383069 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.383148 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.385098 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.385551 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.386452 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.386637 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.386694 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.389172 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.389173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.389174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.389181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.391663 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.392032 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.392033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.392033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.392087 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.392274 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.392275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.392275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.392386 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.392426 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.392700 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.392761 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:16:55.393031 7585 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43625
I20260620 09:16:55.393123 7585 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:55.393220 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.382227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37499 (local address 127.0.98.2:45817)
0620 09:16:55.382379 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.382413 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.382931 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.383222 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.383225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.383664 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.383874 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.383879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.385627 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.385628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.386376 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.386379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.386697 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.386698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.386769 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.387124 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.387129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.389092 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.391736 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.391737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.391737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.391915 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.392156 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.392157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.392157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.392210 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.392468 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.392469 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.392509 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.393015 (+ 506us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.393102 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260620 09:16:55.393311 7585 heartbeater.cc:511] Master 127.0.98.60:43625 requested a full tablet report, sending...
I20260620 09:16:55.393740 7268 ts_manager.cc:194] Registered new tserver with Master: fbc5044d7eec4e3db65214b485134ca8 (127.0.98.2:40687)
I20260620 09:16:55.394466 7587 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37499
I20260620 09:16:55.394467 7586 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41791
I20260620 09:16:55.394532 7587 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:55.394550 7586 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:55.394593 7587 heartbeater.cc:511] Master 127.0.98.61:37499 requested a full tablet report, sending...
I20260620 09:16:55.394610 7586 heartbeater.cc:511] Master 127.0.98.62:41791 requested a full tablet report, sending...
I20260620 09:16:55.394847 7082 ts_manager.cc:194] Registered new tserver with Master: fbc5044d7eec4e3db65214b485134ca8 (127.0.98.2:40687)
I20260620 09:16:55.395413 6484 ts_manager.cc:194] Re-registered known tserver with Master: fbc5044d7eec4e3db65214b485134ca8 (127.0.98.2:40687)
I20260620 09:16:55.395823 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44625
I20260620 09:16:55.395869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39167
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41791
--webserver_port=41673
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41791,127.0.98.61:37499
--tserver_master_addrs=127.0.98.62:41791,127.0.98.61:37499,127.0.98.60:43625 with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:55.474587 7592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:55.474786 7592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:55.474817 7592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:55.475943 7592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:55.475981 7592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:55.476015 7592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:55.477427 7592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41673
--tserver_master_addrs=127.0.98.62:41791,127.0.98.61:37499,127.0.98.60:43625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:55.477654 7592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:55.477859 7592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:55.482743 7597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:55.482748 7598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:55.482748 7600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:55.482836 7592 server_base.cc:1034] running on GCE node
I20260620 09:16:55.483217 7592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:55.483484 7592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:55.484650 7592 hybrid_clock.cc:648] HybridClock initialized: now 1781947015484628 us; error 37 us; skew 500 ppm
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:55.487018 7592 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 09:16:55.487442 7592 webserver.cc:460] Webserver started at http://127.0.98.3:41673/ using document root <none> and password file <none>
I20260620 09:16:55.487691 7592 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:55.487756 7592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:55.489046 7592 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:55.489636 7607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.489797 7592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:55.489874 7592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "ec6b0b7db96744d3919456e1c5f5cc39"
format_stamp: "Formatted at 2026-06-20 09:16:54 on dist-test-slave-rq8c"
server_key: "092321c88e141d12061694e3fc57cc6f"
server_key_iv: "f3a4ac86189abb7df97c7d9f92c78ebe"
server_key_version: "encryptionkey@0"
I20260620 09:16:55.490199 7592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:55.504200 7592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:55.505868 7592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:55.505988 7592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:55.506181 7592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:16:55.506431 7592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:55.506462 7592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.506489 7592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:55.506512 7592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:55.511706 7592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39167
I20260620 09:16:55.511744 7720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39167 every 8 connection(s)
I20260620 09:16:55.512053 7592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 09:16:55 dist-test-slave-rq8c krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947015, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:16:55.520247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 7592
I20260620 09:16:55.522751 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.513068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46159 (local address 127.0.98.62:41791)
0620 09:16:55.513115 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.513117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.513169 (+ 52us) server_negotiation.cc:407] Connection header received
0620 09:16:55.514150 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.514151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.514161 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.514212 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.514763 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.515266 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.516505 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.516814 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.516867 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.519083 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.519084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.519084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.519101 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.520751 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.521664 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.521665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.521666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.521723 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.521981 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.521982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.521983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.522130 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.522178 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.522568 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.522658 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:16:55.523270 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.512961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.3:46159)
0620 09:16:55.513127 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.513135 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.513880 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.514228 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.514229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.514477 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.514662 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.514668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.515364 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.515365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.516425 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.516427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.516923 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.516924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.516990 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.517465 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.517475 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.518975 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.520820 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.520827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.520829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.521569 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.521792 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.521792 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.521793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.521910 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.522214 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.522220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.522385 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.522866 (+ 481us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.523050 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":5,"thread_start_us":52,"threads_started":1}
I20260620 09:16:55.523355 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.512671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55123 (local address 127.0.98.61:37499)
0620 09:16:55.512764 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.512766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.513012 (+ 246us) server_negotiation.cc:407] Connection header received
0620 09:16:55.514025 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.514027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.514038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.514097 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.514736 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.515526 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.516457 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.516763 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.516818 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.520006 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.520007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.520008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.520015 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.522174 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.522591 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.522592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.522592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.522655 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.522842 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.522843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.522843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.522963 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.523003 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.523190 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.523260 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 09:16:55.523643 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.512692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37499 (local address 127.0.98.3:55123)
0620 09:16:55.512901 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.512940 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.513863 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.514113 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.514116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.514447 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.514661 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.514666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.515666 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.515667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.516379 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.516381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.516843 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.516845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.516989 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.517686 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.517690 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.519927 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.522246 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.522247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.522248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.522504 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.522722 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.522723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.522723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.522776 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.523048 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.523049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.523086 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.523435 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.523526 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260620 09:16:55.523787 7722 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41791
I20260620 09:16:55.523895 7722 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:55.523916 7723 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37499
I20260620 09:16:55.523963 7723 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:55.524066 7723 heartbeater.cc:511] Master 127.0.98.61:37499 requested a full tablet report, sending...
I20260620 09:16:55.524066 7722 heartbeater.cc:511] Master 127.0.98.62:41791 requested a full tablet report, sending...
I20260620 09:16:55.524456 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.512693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43625 (local address 127.0.98.3:36055)
0620 09:16:55.512940 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.512949 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.513870 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.514112 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.514115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.514477 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:55.514662 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.514668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.515577 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.515578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.516458 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.516459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.516910 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.516911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.516989 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.517765 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:55.517769 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:55.519848 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:55.522268 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.522269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.522269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.522533 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.523027 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:55.523027 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:55.523028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:55.523093 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:55.524197 (+ 1104us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:55.524197 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:55.524246 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.524305 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.524391 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":23,"thread_start_us":92,"threads_started":1}
I20260620 09:16:55.524463 7082 ts_manager.cc:194] Registered new tserver with Master: ec6b0b7db96744d3919456e1c5f5cc39 (127.0.98.3:39167)
I20260620 09:16:55.524667 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.512694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36055 (local address 127.0.98.60:43625)
0620 09:16:55.512751 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.512753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.513034 (+ 281us) server_negotiation.cc:407] Connection header received
0620 09:16:55.513956 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.513957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.513969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.514030 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.514771 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.515504 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.516561 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.516807 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.516862 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.519968 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.519970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.519970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.519978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.521879 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.522703 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.522704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.522704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.522768 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.523157 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.523158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.523159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.523276 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.523328 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.524523 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.524577 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:16:55.525056 7721 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43625
I20260620 09:16:55.525097 7721 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:55.525105 6484 ts_manager.cc:194] Re-registered known tserver with Master: ec6b0b7db96744d3919456e1c5f5cc39 (127.0.98.3:39167)
I20260620 09:16:55.525159 7721 heartbeater.cc:511] Master 127.0.98.60:43625 requested a full tablet report, sending...
I20260620 09:16:55.525460 7268 ts_manager.cc:194] Registered new tserver with Master: ec6b0b7db96744d3919456e1c5f5cc39 (127.0.98.3:39167)
I20260620 09:16:55.525543 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46159
I20260620 09:16:55.529542 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.521239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.0.98.62:41791)
0620 09:16:55.521359 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.521360 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.521516 (+ 156us) server_negotiation.cc:407] Connection header received
0620 09:16:55.521609 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.521609 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.521621 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.521662 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.522460 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.523254 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.524288 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.524549 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.524609 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.526612 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.526614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.526614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.526620 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.528247 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.528533 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.528533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.528534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.528590 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.528803 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.528804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.528804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.528931 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.529019 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.529360 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.529438 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
I20260620 09:16:55.530791 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.521501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.0.98.60:43625)
0620 09:16:55.524726 (+ 3225us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.524728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.524735 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:55.524750 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.524751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.524760 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.524816 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.525497 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.526165 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.527013 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.527244 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.527296 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.528385 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.528387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.528387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.528393 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.529732 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.530035 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.530036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.530036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.530085 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.530242 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.530243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.530244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.530376 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.530455 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.530640 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.530701 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3213,"thread_start_us":47,"threads_started":1}
I20260620 09:16:55.532651 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.525234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.0.98.61:37499)
0620 09:16:55.525551 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.525552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.525558 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:16:55.525574 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.525575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.525583 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.525628 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.527075 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.527795 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.528708 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.528929 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.528984 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.530279 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.530280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.530280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.530287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.531715 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.532035 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.532036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.532036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.532089 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.532252 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.532253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.532253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.532349 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.532416 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.532523 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.532594 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260620 09:16:55.531399 6484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:55.533392 6484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:55.540385 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.536721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39167 (local address 127.0.0.1:43448)
0620 09:16:55.536833 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.536841 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.536874 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.537238 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.537239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.537241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:55.537295 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.537296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.538322 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.538323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.539408 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.539410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.540175 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.540176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.540233 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.540236 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.540277 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.540317 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260620 09:16:55.540778 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.536763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46105 (local address 127.0.0.1:40738)
0620 09:16:55.536969 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.536979 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.537026 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.537639 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.537640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.537642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:55.537686 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.537687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.538791 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.538792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.539648 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.539651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.540550 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.540551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.540610 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.540614 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.540677 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.540716 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260620 09:16:55.540808 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.536945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.0.98.1:46105)
0620 09:16:55.537448 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.537452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.537461 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:16:55.537534 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.537536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.537550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.537623 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:55.537748 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.538726 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.539730 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.540468 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.540544 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.540615 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.540694 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.540732 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":70,"threads_started":1}
I20260620 09:16:55.540863 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.536679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40687 (local address 127.0.0.1:44676)
0620 09:16:55.536713 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.536723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.536760 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.537339 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.537340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.537341 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:55.537367 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.537368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.538595 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.538596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.539600 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.539602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.540675 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.540676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.540737 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.540740 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.540768 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.540808 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 09:16:55.541225 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.536912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.0.98.3:39167)
0620 09:16:55.537064 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.537069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.537079 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:16:55.537114 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.537116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.537132 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.537242 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:55.537365 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.538249 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.539517 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.540102 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.540164 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.540774 (+ 610us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.540804 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.541154 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260620 09:16:55.541605 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.536775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44676 (local address 127.0.98.2:40687)
0620 09:16:55.537149 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.537153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.537162 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:16:55.537199 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.537201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.537216 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.537329 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:55.537433 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.538521 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.539681 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.540592 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.541006 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.541063 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.541083 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.541114 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":64,"threads_started":1}
I20260620 09:16:55.541741 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2465280c874e3bb79c7a6f3f490de6 (DEFAULT_TABLE table=test-table [id=13f7e172da664820808c8921ae4d4cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:55.541806 7655 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2465280c874e3bb79c7a6f3f490de6 (DEFAULT_TABLE table=test-table [id=13f7e172da664820808c8921ae4d4cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:55.542018 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b2465280c874e3bb79c7a6f3f490de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:55.542048 7655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b2465280c874e3bb79c7a6f3f490de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:55.544255 7519 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2465280c874e3bb79c7a6f3f490de6 (DEFAULT_TABLE table=test-table [id=13f7e172da664820808c8921ae4d4cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:55.544579 7519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b2465280c874e3bb79c7a6f3f490de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:55.544703 7746 tablet_bootstrap.cc:492] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c: Bootstrap starting.
I20260620 09:16:55.544930 7747 tablet_bootstrap.cc:492] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39: Bootstrap starting.
I20260620 09:16:55.545480 7746 tablet_bootstrap.cc:654] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:55.545828 7746 log.cc:824] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:55.546080 7747 tablet_bootstrap.cc:654] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:55.546372 7747 log.cc:824] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:55.546651 7746 tablet_bootstrap.cc:492] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c: No bootstrap required, opened a new log
I20260620 09:16:55.546707 7746 ts_tablet_manager.cc:1397] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:55.547065 7747 tablet_bootstrap.cc:492] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39: No bootstrap required, opened a new log
I20260620 09:16:55.547129 7747 ts_tablet_manager.cc:1397] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:16:55.547761 7750 tablet_bootstrap.cc:492] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8: Bootstrap starting.
I20260620 09:16:55.548260 7750 tablet_bootstrap.cc:654] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:55.548528 7750 log.cc:824] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:55.548883 7746 raft_consensus.cc:348] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } }
I20260620 09:16:55.548900 7747 raft_consensus.cc:348] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } }
I20260620 09:16:55.549048 7746 raft_consensus.cc:374] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:55.549048 7747 raft_consensus.cc:374] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:55.549081 7746 raft_consensus.cc:729] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8587525e365e445c8df92975c7dbd21c, State: Initialized, Role: FOLLOWER
I20260620 09:16:55.549081 7747 raft_consensus.cc:729] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6b0b7db96744d3919456e1c5f5cc39, State: Initialized, Role: FOLLOWER
I20260620 09:16:55.549147 7750 tablet_bootstrap.cc:492] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8: No bootstrap required, opened a new log
I20260620 09:16:55.549199 7750 ts_tablet_manager.cc:1397] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:55.549193 7746 consensus_queue.cc:260] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } }
I20260620 09:16:55.549245 7747 consensus_queue.cc:260] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } }
I20260620 09:16:55.549455 7747 ts_tablet_manager.cc:1428] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:16:55.549455 7746 ts_tablet_manager.cc:1428] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:16:55.549731 7450 heartbeater.cc:503] Master 127.0.98.62:41791 was elected leader, sending a full tablet report...
I20260620 09:16:55.549743 7722 heartbeater.cc:503] Master 127.0.98.62:41791 was elected leader, sending a full tablet report...
I20260620 09:16:55.552062 7750 raft_consensus.cc:348] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } }
I20260620 09:16:55.552182 7750 raft_consensus.cc:374] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:55.552212 7750 raft_consensus.cc:729] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc5044d7eec4e3db65214b485134ca8, State: Initialized, Role: FOLLOWER
I20260620 09:16:55.552321 7750 consensus_queue.cc:260] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } }
I20260620 09:16:55.552541 7750 ts_tablet_manager.cc:1428] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:16:55.552611 7586 heartbeater.cc:503] Master 127.0.98.62:41791 was elected leader, sending a full tablet report...
I20260620 09:16:55.588542 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.586177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.0.98.60:43625)
0620 09:16:55.586228 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.586230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.586238 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:55.586262 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.586263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.586281 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.586335 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:55.586448 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.586929 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.587843 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.588234 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.588293 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.588377 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.588429 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.588466 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:16:55.588542 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.586082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43625 (local address 127.0.0.1:53362)
0620 09:16:55.586145 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.586156 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.586200 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.586344 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.586345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.586346 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:55.586398 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.586400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.587004 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.587005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.587767 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.587769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.588308 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.588309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.588367 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.588371 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.588423 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.588468 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 09:16:55.588955 7278 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260620 09:16:55.589792 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4bd2c7b810f4490a53b96a66bd7a3ca. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.589912 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: This master's current role is: LEARNER
W20260620 09:16:55.632261 7588 tablet.cc:2369] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:16:55.755177 7452 tablet.cc:2369] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:16:55.762980 7724 tablet.cc:2369] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:16:55.873567 7754 raft_consensus.cc:482] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:55.873726 7754 raft_consensus.cc:504] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } }
I20260620 09:16:55.874125 7754 leader_election.cc:288] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec6b0b7db96744d3919456e1c5f5cc39 (127.0.98.3:39167), 8587525e365e445c8df92975c7dbd21c (127.0.98.1:46105)
I20260620 09:16:55.876839 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.874335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39167 (local address 127.0.98.2:39499)
0620 09:16:55.874397 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.874410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.874478 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.874634 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.874635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.874637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:55.874702 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.874703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.875307 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.875308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.876101 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.876104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.876609 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.876610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.876671 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.876674 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.876720 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.876759 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 09:16:55.876839 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.874460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39499 (local address 127.0.98.3:39167)
0620 09:16:55.874515 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.874517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.874524 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:55.874552 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.874553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.874562 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.874607 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:55.874757 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.875229 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.876187 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.876534 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.876594 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.876604 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.876735 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.876775 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:16:55.877132 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.874596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46105 (local address 127.0.98.2:34029)
0620 09:16:55.874645 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:55.874653 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:55.874702 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:55.874826 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:55.874826 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:55.874828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:55.874889 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.874890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.875577 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.875578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.876382 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:55.876384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.876875 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.876876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:55.876958 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.876961 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:55.877011 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 09:16:55.877051 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 09:16:55.877185 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.874444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34029 (local address 127.0.98.1:46105)
0620 09:16:55.874501 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.874503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.874702 (+ 199us) server_negotiation.cc:407] Connection header received
0620 09:16:55.874764 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.874765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.874776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.874818 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:55.874941 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.875471 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.876461 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.876809 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.876868 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.876877 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.877076 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.877119 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:16:55.877224 7675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2465280c874e3bb79c7a6f3f490de6" candidate_uuid: "fbc5044d7eec4e3db65214b485134ca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" is_pre_election: true
I20260620 09:16:55.877400 7675 raft_consensus.cc:2454] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc5044d7eec4e3db65214b485134ca8 in term 0.
I20260620 09:16:55.877473 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2465280c874e3bb79c7a6f3f490de6" candidate_uuid: "fbc5044d7eec4e3db65214b485134ca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8587525e365e445c8df92975c7dbd21c" is_pre_election: true
I20260620 09:16:55.877624 7403 raft_consensus.cc:2454] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc5044d7eec4e3db65214b485134ca8 in term 0.
I20260620 09:16:55.877635 7474 leader_election.cc:302] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec6b0b7db96744d3919456e1c5f5cc39, fbc5044d7eec4e3db65214b485134ca8; no voters:
I20260620 09:16:55.877795 7754 raft_consensus.cc:2781] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:55.877849 7754 raft_consensus.cc:482] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:55.877864 7754 raft_consensus.cc:3037] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:55.878528 7754 raft_consensus.cc:504] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } }
I20260620 09:16:55.878645 7754 leader_election.cc:288] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [CANDIDATE]: Term 1 election: Requested vote from peers ec6b0b7db96744d3919456e1c5f5cc39 (127.0.98.3:39167), 8587525e365e445c8df92975c7dbd21c (127.0.98.1:46105)
I20260620 09:16:55.878798 7675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2465280c874e3bb79c7a6f3f490de6" candidate_uuid: "fbc5044d7eec4e3db65214b485134ca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6b0b7db96744d3919456e1c5f5cc39"
I20260620 09:16:55.878826 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2465280c874e3bb79c7a6f3f490de6" candidate_uuid: "fbc5044d7eec4e3db65214b485134ca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8587525e365e445c8df92975c7dbd21c"
I20260620 09:16:55.878865 7675 raft_consensus.cc:3037] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:55.878895 7403 raft_consensus.cc:3037] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:55.879521 7675 raft_consensus.cc:2454] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc5044d7eec4e3db65214b485134ca8 in term 1.
I20260620 09:16:55.879679 7403 raft_consensus.cc:2454] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc5044d7eec4e3db65214b485134ca8 in term 1.
I20260620 09:16:55.879685 7474 leader_election.cc:302] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec6b0b7db96744d3919456e1c5f5cc39, fbc5044d7eec4e3db65214b485134ca8; no voters:
I20260620 09:16:55.879781 7754 raft_consensus.cc:2781] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:55.879926 7754 raft_consensus.cc:686] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 1 LEADER]: Becoming Leader. State: Replica: fbc5044d7eec4e3db65214b485134ca8, State: Running, Role: LEADER
I20260620 09:16:55.880028 7754 consensus_queue.cc:237] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } } peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } }
I20260620 09:16:55.880867 6484 catalog_manager.cc:5495] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbc5044d7eec4e3db65214b485134ca8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fbc5044d7eec4e3db65214b485134ca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } health_report { overall_health: UNKNOWN } } }
I20260620 09:16:55.881381 7232 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: true } } }
I20260620 09:16:55.881467 7232 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:55.882058 7043 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca: attempting to promote NON_VOTER 83aae58fc5f546a9a9b928944595c5c5 to VOTER
I20260620 09:16:55.882193 7043 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } }
I20260620 09:16:55.882663 7278 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 1 LEARNER]: Refusing update from remote peer a4bd2c7b810f4490a53b96a66bd7a3ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 09:16:55.882727 7092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 FOLLOWER]: Refusing update from remote peer a4bd2c7b810f4490a53b96a66bd7a3ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 09:16:55.882877 7232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 09:16:55.882958 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 09:16:55.883490 7044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 83aae58fc5f546a9a9b928944595c5c5 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } } }
I20260620 09:16:55.883605 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 83aae58fc5f546a9a9b928944595c5c5 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } } }
I20260620 09:16:55.883774 7278 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 83aae58fc5f546a9a9b928944595c5c5 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } } }
I20260620 09:16:55.884392 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } } }
I20260620 09:16:55.884466 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:55.884534 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } } }
I20260620 09:16:55.884583 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:55.884626 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } } }
I20260620 09:16:55.884474 7232 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } } }
I20260620 09:16:55.884708 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:16:55.884718 7232 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:55.884752 7761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } } }
I20260620 09:16:55.884831 7761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:16:55.890617 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.888474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.0.98.2:40687)
0620 09:16:55.888530 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.888533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.888541 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:55.888588 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.888588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.888606 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.888663 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:55.888761 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.889216 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.889884 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.890083 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.890139 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.890309 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:55.890407 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:55.890463 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.890537 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.890555 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:16:55.892370 7675 raft_consensus.cc:1261] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39 [term 1 FOLLOWER]: Refusing update from remote peer fbc5044d7eec4e3db65214b485134ca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:55.892412 7403 raft_consensus.cc:1261] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c [term 1 FOLLOWER]: Refusing update from remote peer fbc5044d7eec4e3db65214b485134ca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:55.892695 7758 consensus_queue.cc:1035] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:55.892833 7754 consensus_queue.cc:1035] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:55.896544 7765 mvcc.cc:205] Tried to move back new op lower bound from 7298854977092718592 to 7298854977045344256. Current Snapshot: MvccSnapshot[applied={T|T < 7298854977092718592 or (T in {7298854977092718592})}]
I20260620 09:16:55.896778 7764 mvcc.cc:205] Tried to move back new op lower bound from 7298854977092718592 to 7298854977045344256. Current Snapshot: MvccSnapshot[applied={T|T < 7298854977092718592}]
I20260620 09:16:55.898696 6484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:55.898773 6484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:55.899962 6484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4bd2c7b810f4490a53b96a66bd7a3ca: Sending DeleteTablet for 3 replicas of tablet 3b2465280c874e3bb79c7a6f3f490de6
I20260620 09:16:55.900344 7519 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b2465280c874e3bb79c7a6f3f490de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44676
I20260620 09:16:55.900377 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b2465280c874e3bb79c7a6f3f490de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40738
I20260620 09:16:55.900524 7655 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b2465280c874e3bb79c7a6f3f490de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43448
I20260620 09:16:55.900734 7774 tablet_replica.cc:330] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39: stopping tablet replica
I20260620 09:16:55.900835 7774 raft_consensus.cc:2229] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:55.900909 7772 tablet_replica.cc:330] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8: stopping tablet replica
I20260620 09:16:55.900945 7774 raft_consensus.cc:2258] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:55.901152 7772 raft_consensus.cc:2229] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:55.901314 7772 raft_consensus.cc:2258] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:55.901379 7774 ts_tablet_manager.cc:1905] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:55.901630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 6452
I20260620 09:16:55.902140 7772 ts_tablet_manager.cc:1905] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:55.903002 7774 ts_tablet_manager.cc:1918] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:55.903070 7774 log.cc:1196] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/3b2465280c874e3bb79c7a6f3f490de6
I20260620 09:16:55.903321 7774 ts_tablet_manager.cc:1939] T 3b2465280c874e3bb79c7a6f3f490de6 P ec6b0b7db96744d3919456e1c5f5cc39: Deleting consensus metadata
I20260620 09:16:55.904071 7772 ts_tablet_manager.cc:1918] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:55.904145 7772 log.cc:1196] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/3b2465280c874e3bb79c7a6f3f490de6
I20260620 09:16:55.904392 7772 ts_tablet_manager.cc:1939] T 3b2465280c874e3bb79c7a6f3f490de6 P fbc5044d7eec4e3db65214b485134ca8: Deleting consensus metadata
W20260620 09:16:55.907049 7474 connection.cc:535] server connection from 127.0.0.1:44676 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 09:16:55.907058 7609 connection.cc:535] server connection from 127.0.0.1:43448 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:16:55.907714 7773 tablet_replica.cc:330] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c: stopping tablet replica
I20260620 09:16:55.907820 7773 raft_consensus.cc:2229] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:55.907928 7773 raft_consensus.cc:2258] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:55.908313 7773 ts_tablet_manager.cc:1905] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:55.909811 7773 ts_tablet_manager.cc:1918] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:55.909875 7773 log.cc:1196] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-0/wal/wals/3b2465280c874e3bb79c7a6f3f490de6
I20260620 09:16:55.910101 7773 ts_tablet_manager.cc:1939] T 3b2465280c874e3bb79c7a6f3f490de6 P 8587525e365e445c8df92975c7dbd21c: Deleting consensus metadata
W20260620 09:16:55.910383 7339 connection.cc:441] server connection from 127.0.0.1:40738 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40738 (request call id 1) could send its response
I20260620 09:16:55.915550 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.908566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33978 (local address 127.0.98.61:37499)
0620 09:16:55.908605 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.908607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.908861 (+ 254us) server_negotiation.cc:407] Connection header received
0620 09:16:55.908958 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.908959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.908969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.908998 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.909683 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.910419 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.911146 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.911339 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.911395 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.912900 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.912901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.912902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.912908 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.914389 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.914742 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.914743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.914743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.914800 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.914987 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.914988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.914988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.915140 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.915226 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.915407 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.915480 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:16:55.916023 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:55.908860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.0.98.60:43625)
0620 09:16:55.908900 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:55.908902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:55.909147 (+ 245us) server_negotiation.cc:407] Connection header received
0620 09:16:55.909262 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:55.909263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:55.909273 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:55.909299 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:55.909958 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.910725 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.911779 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:55.912030 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:55.912087 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:55.913354 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:55.913355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:55.913355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:55.913362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:55.914865 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.915191 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.915192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.915192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.915252 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:55.915442 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:55.915443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:55.915443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:55.915560 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:55.915662 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:55.915843 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 09:16:55.915918 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:16:56.897037 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:56.896548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.3:57209)
0620 09:16:56.896885 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:56.896965 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":60,"threads_started":1}
I20260620 09:16:56.897190 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:56.896754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.2:33365)
0620 09:16:56.897062 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:56.897143 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":46,"threads_started":1}
W20260620 09:16:56.897248 7722 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41791 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
I20260620 09:16:56.897369 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:56.897066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.1:44865)
0620 09:16:56.897271 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:56.897332 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":58,"threads_started":1}
W20260620 09:16:56.897405 7586 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41791 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
W20260620 09:16:56.897627 7450 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41791 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
I20260620 09:16:57.400437 7785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4bd2c7b810f4490a53b96a66bd7a3ca)
I20260620 09:16:57.400589 7785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } }
I20260620 09:16:57.400841 7785 leader_election.cc:288] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4bd2c7b810f4490a53b96a66bd7a3ca (127.0.98.62:41791), 83aae58fc5f546a9a9b928944595c5c5 (127.0.98.60:43625)
I20260620 09:16:57.401088 7786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4bd2c7b810f4490a53b96a66bd7a3ca)
I20260620 09:16:57.401162 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.400971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:42080)
0620 09:16:57.401072 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.401122 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
W20260620 09:16:57.401314 7066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
I20260620 09:16:57.401342 7786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } }
I20260620 09:16:57.401701 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.401619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:42086)
0620 09:16:57.401649 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.401675 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 09:16:57.401824 7066 leader_election.cc:334] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4bd2c7b810f4490a53b96a66bd7a3ca (127.0.98.62:41791): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
I20260620 09:16:57.401937 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.401734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:42096)
0620 09:16:57.401855 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.401894 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":43,"threads_started":1}
I20260620 09:16:57.402031 7786 leader_election.cc:288] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4bd2c7b810f4490a53b96a66bd7a3ca (127.0.98.62:41791), 04208839d9c74e7eb1b9db21c7cf51ea (127.0.98.61:37499)
W20260620 09:16:57.402098 7252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
I20260620 09:16:57.402453 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.402378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:42108)
0620 09:16:57.402404 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.402429 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 09:16:57.402577 7252 leader_election.cc:334] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4bd2c7b810f4490a53b96a66bd7a3ca (127.0.98.62:41791): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
I20260620 09:16:57.404417 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.401302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.0.98.60:43625)
0620 09:16:57.401440 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:57.401442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:57.401452 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:16:57.401478 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:57.401479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:57.401489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:57.401566 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:57.401730 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.402401 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.403732 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.404121 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.404178 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.404189 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:57.404315 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 09:16:57.404353 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":115,"thread_start_us":50,"threads_started":1}
I20260620 09:16:57.404421 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.401137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43625 (local address 127.0.0.1:52118)
0620 09:16:57.401223 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.401232 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:57.401387 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:57.401587 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:57.401588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:57.401590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:57.401651 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.401654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.402514 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.402515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.403674 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.403677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.404189 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.404190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.404257 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.404261 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:57.404305 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 09:16:57.404350 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":41,"threads_started":1}
I20260620 09:16:57.404651 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "83aae58fc5f546a9a9b928944595c5c5" is_pre_election: true
I20260620 09:16:57.404750 7278 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04208839d9c74e7eb1b9db21c7cf51ea in term 1.
I20260620 09:16:57.404881 7066 leader_election.cc:302] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04208839d9c74e7eb1b9db21c7cf51ea, 83aae58fc5f546a9a9b928944595c5c5; no voters: a4bd2c7b810f4490a53b96a66bd7a3ca
I20260620 09:16:57.405014 7785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 09:16:57.405047 7785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4bd2c7b810f4490a53b96a66bd7a3ca)
I20260620 09:16:57.405071 7785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:16:57.405176 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.401915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33984 (local address 127.0.98.61:37499)
0620 09:16:57.402154 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:57.402156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:57.402164 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:57.402446 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:57.402447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:57.402457 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:57.402519 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:57.402665 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.403375 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.404350 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.404863 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.404952 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.404964 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:57.405085 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 09:16:57.405120 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
I20260620 09:16:57.405176 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.401778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37499 (local address 127.0.0.1:33984)
0620 09:16:57.401957 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.401966 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:57.402275 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:57.402563 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:57.402564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:57.402566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:57.402604 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.402606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.403451 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.403452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.404271 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.404273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.404953 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.404954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.405030 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.405033 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:57.405076 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 09:16:57.405113 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":2,"thread_start_us":45,"threads_started":1}
I20260620 09:16:57.405385 7092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83aae58fc5f546a9a9b928944595c5c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" is_pre_election: true
I20260620 09:16:57.405761 7785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } }
I20260620 09:16:57.405857 7092 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83aae58fc5f546a9a9b928944595c5c5 in current term 2: Already voted for candidate 04208839d9c74e7eb1b9db21c7cf51ea in this term.
I20260620 09:16:57.405859 7785 leader_election.cc:288] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [CANDIDATE]: Term 2 election: Requested vote from peers a4bd2c7b810f4490a53b96a66bd7a3ca (127.0.98.62:41791), 83aae58fc5f546a9a9b928944595c5c5 (127.0.98.60:43625)
I20260620 09:16:57.406051 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.405946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:42116)
0620 09:16:57.406008 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.406030 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 09:16:57.406054 7254 leader_election.cc:302] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83aae58fc5f546a9a9b928944595c5c5; no voters: 04208839d9c74e7eb1b9db21c7cf51ea, a4bd2c7b810f4490a53b96a66bd7a3ca
I20260620 09:16:57.406208 7786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:16:57.406211 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "83aae58fc5f546a9a9b928944595c5c5"
I20260620 09:16:57.406821 7786 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:16:57.407124 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.407041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:42126)
0620 09:16:57.407081 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.407107 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 09:16:57.407191 7066 leader_election.cc:334] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4bd2c7b810f4490a53b96a66bd7a3ca (127.0.98.62:41791): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
I20260620 09:16:57.407570 7278 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04208839d9c74e7eb1b9db21c7cf51ea in term 2.
I20260620 09:16:57.407788 7066 leader_election.cc:302] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04208839d9c74e7eb1b9db21c7cf51ea, 83aae58fc5f546a9a9b928944595c5c5; no voters: a4bd2c7b810f4490a53b96a66bd7a3ca
I20260620 09:16:57.407869 7785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:16:57.407989 7785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [term 2 LEADER]: Becoming Leader. State: Replica: 04208839d9c74e7eb1b9db21c7cf51ea, State: Running, Role: LEADER
I20260620 09:16:57.408046 7785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } }
I20260620 09:16:57.408299 7795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04208839d9c74e7eb1b9db21c7cf51ea. Latest consensus state: current_term: 2 leader_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } } }
I20260620 09:16:57.408413 7795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:57.408599 7797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:57.409008 7797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:57.409066 7797 catalog_manager.cc:1259] Loaded cluster ID: dbf97b58aadf4ef59ca61a172f58a443
I20260620 09:16:57.409096 7797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:16:57.409241 7797 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:57.409304 7797 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: Loaded TSK: 0
I20260620 09:16:57.409435 7797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:16:57.446143 7082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:57.447666 7082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:57.448571 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.448471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:42132)
0620 09:16:57.448511 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.448539 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 09:16:57.448660 7278 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [term 2 FOLLOWER]: Refusing update from remote peer 04208839d9c74e7eb1b9db21c7cf51ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 09:16:57.448874 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.448781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.0.1:42136)
0620 09:16:57.448818 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.448848 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 09:16:57.448885 7785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 09:16:57.449008 7066 consensus_peers.cc:487] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea -> Peer a4bd2c7b810f4490a53b96a66bd7a3ca (127.0.98.62:41791): Couldn't send request to peer a4bd2c7b810f4490a53b96a66bd7a3ca. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 09:16:57.449484 7786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04208839d9c74e7eb1b9db21c7cf51ea. Latest consensus state: current_term: 2 leader_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } } }
I20260620 09:16:57.449573 7786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:16:57.449980 7785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } } }
I20260620 09:16:57.450062 7785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:57.450228 7786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a4bd2c7b810f4490a53b96a66bd7a3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41791 } } peers { permanent_uuid: "04208839d9c74e7eb1b9db21c7cf51ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37499 } attrs { promote: false } } peers { permanent_uuid: "83aae58fc5f546a9a9b928944595c5c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43625 } attrs { promote: false } } }
I20260620 09:16:57.450311 7786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aae58fc5f546a9a9b928944595c5c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:16:57.450379 7799 mvcc.cc:205] Tried to move back new op lower bound from 7298854983467720704 to 7298854983303798784. Current Snapshot: MvccSnapshot[applied={T|T < 7298854983467720704}]
I20260620 09:16:57.454224 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.451418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46105 (local address 127.0.0.1:37560)
0620 09:16:57.451522 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.451532 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:57.451569 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:57.451876 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:57.451877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:57.451879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:57.451921 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.451923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.452587 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.452588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.453498 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.453500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.454010 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.454011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.454071 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.454075 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:57.454125 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 09:16:57.454162 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 09:16:57.454429 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.451649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.0.98.1:46105)
0620 09:16:57.451756 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:57.451757 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:57.451765 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:57.451790 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:57.451791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:57.451802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:57.451860 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:57.451994 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.452519 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.453575 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.453929 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.454001 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.454014 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:57.454143 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 09:16:57.454191 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":58,"threads_started":1}
I20260620 09:16:57.454764 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 7f898b30117a4821b39a37c877e3c9b4 (DEFAULT_TABLE table=test-table [id=41b2bccde9e644a49e09d16ee1b98e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:57.454901 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f898b30117a4821b39a37c877e3c9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:57.454959 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.451502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40687 (local address 127.0.0.1:35442)
0620 09:16:57.451608 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.451636 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:57.451683 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:57.452025 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:57.452026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:57.452028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:57.452074 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.452076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.452980 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.452981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.453961 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.453963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.454738 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.454739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.454796 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.454800 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:57.454846 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 09:16:57.454895 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260620 09:16:57.455204 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.451789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.0.98.2:40687)
0620 09:16:57.451882 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:57.451884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:57.451891 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:57.451914 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:57.451915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:57.451926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:57.452134 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:57.452171 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.452903 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.454049 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.454576 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.454638 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.454651 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:57.455096 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 09:16:57.455142 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":48,"threads_started":1}
I20260620 09:16:57.455821 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.451803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.0.98.3:39167)
0620 09:16:57.451948 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:57.451950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:57.451958 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:57.451978 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:57.451980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:57.451991 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:57.452151 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:57.452185 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.452909 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.455015 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.455508 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.455566 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.455576 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:57.455741 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 09:16:57.455771 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":58,"threads_started":1}
I20260620 09:16:57.455832 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.451494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39167 (local address 127.0.0.1:36038)
0620 09:16:57.451531 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.451538 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:57.451573 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:57.452069 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:57.452070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:57.452071 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:57.452098 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.452099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.453003 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.453004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.454918 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.454921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.455589 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.455590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.455675 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.455678 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:57.455738 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:16:57.455775 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":941}
I20260620 09:16:57.456147 7655 tablet_service.cc:1467] Processing CreateTablet for tablet 7f898b30117a4821b39a37c877e3c9b4 (DEFAULT_TABLE table=test-table [id=41b2bccde9e644a49e09d16ee1b98e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:57.456241 7807 tablet_bootstrap.cc:492] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c: Bootstrap starting.
I20260620 09:16:57.456259 7655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f898b30117a4821b39a37c877e3c9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:57.456434 7519 tablet_service.cc:1467] Processing CreateTablet for tablet 7f898b30117a4821b39a37c877e3c9b4 (DEFAULT_TABLE table=test-table [id=41b2bccde9e644a49e09d16ee1b98e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:57.456542 7519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f898b30117a4821b39a37c877e3c9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:57.456807 7807 tablet_bootstrap.cc:654] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:57.457433 7807 tablet_bootstrap.cc:492] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c: No bootstrap required, opened a new log
I20260620 09:16:57.457492 7807 ts_tablet_manager.cc:1397] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:57.457458 7809 tablet_bootstrap.cc:492] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39: Bootstrap starting.
I20260620 09:16:57.457647 7807 raft_consensus.cc:348] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } } peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } }
I20260620 09:16:57.457729 7807 raft_consensus.cc:374] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:57.457751 7807 raft_consensus.cc:729] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8587525e365e445c8df92975c7dbd21c, State: Initialized, Role: FOLLOWER
I20260620 09:16:57.457795 7807 consensus_queue.cc:260] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } } peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } }
I20260620 09:16:57.457923 7809 tablet_bootstrap.cc:654] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:57.457926 7807 ts_tablet_manager.cc:1428] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:57.457949 7810 tablet_bootstrap.cc:492] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8: Bootstrap starting.
I20260620 09:16:57.458453 7810 tablet_bootstrap.cc:654] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:57.458513 7809 tablet_bootstrap.cc:492] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39: No bootstrap required, opened a new log
I20260620 09:16:57.458561 7809 ts_tablet_manager.cc:1397] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:57.458669 7809 raft_consensus.cc:348] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } } peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } }
I20260620 09:16:57.458755 7809 raft_consensus.cc:374] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:57.458776 7809 raft_consensus.cc:729] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6b0b7db96744d3919456e1c5f5cc39, State: Initialized, Role: FOLLOWER
I20260620 09:16:57.458825 7809 consensus_queue.cc:260] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } } peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } }
I20260620 09:16:57.458846 7451 heartbeater.cc:503] Master 127.0.98.61:37499 was elected leader, sending a full tablet report...
I20260620 09:16:57.459156 7809 ts_tablet_manager.cc:1428] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:57.459420 7810 tablet_bootstrap.cc:492] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8: No bootstrap required, opened a new log
I20260620 09:16:57.459482 7810 ts_tablet_manager.cc:1397] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:57.459534 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.458139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.1:58695)
0620 09:16:57.459439 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.459486 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1275,"thread_start_us":69,"threads_started":1}
I20260620 09:16:57.459596 7810 raft_consensus.cc:348] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } } peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } }
I20260620 09:16:57.459697 7810 raft_consensus.cc:374] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:57.459719 7810 raft_consensus.cc:729] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc5044d7eec4e3db65214b485134ca8, State: Initialized, Role: FOLLOWER
I20260620 09:16:57.459789 7810 consensus_queue.cc:260] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } } peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } }
I20260620 09:16:57.459998 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.459760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.1:51095)
0620 09:16:57.459881 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.459908 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260620 09:16:57.460076 7810 ts_tablet_manager.cc:1428] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:57.460230 7723 heartbeater.cc:503] Master 127.0.98.61:37499 was elected leader, sending a full tablet report...
I20260620 09:16:57.460433 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.459556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.3:59477)
0620 09:16:57.460361 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.460398 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":117,"threads_started":1}
I20260620 09:16:57.460588 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.460346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.2:53031)
0620 09:16:57.460525 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.460558 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20260620 09:16:57.460760 7587 heartbeater.cc:503] Master 127.0.98.61:37499 was elected leader, sending a full tablet report...
I20260620 09:16:57.460958 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.460676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.3:53401)
0620 09:16:57.460895 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.460923 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
I20260620 09:16:57.460968 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.460780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.2:54113)
0620 09:16:57.460898 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.460935 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
I20260620 09:16:57.540292 7815 raft_consensus.cc:482] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:57.540385 7815 raft_consensus.cc:504] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } } peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } }
I20260620 09:16:57.540700 7815 leader_election.cc:288] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8587525e365e445c8df92975c7dbd21c (127.0.98.1:46105), fbc5044d7eec4e3db65214b485134ca8 (127.0.98.2:40687)
I20260620 09:16:57.543419 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.540848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46105 (local address 127.0.98.3:33511)
0620 09:16:57.540900 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.540911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:57.540962 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:57.541128 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:57.541129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:57.541133 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:57.541187 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.541189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.541878 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.541879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.542690 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.542692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.543218 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.543219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.543277 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.543281 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:57.543323 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 09:16:57.543351 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 09:16:57.543442 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.540915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33511 (local address 127.0.98.1:46105)
0620 09:16:57.540968 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:57.540970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:57.540983 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:16:57.541038 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:57.541040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:57.541052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:57.541099 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:57.541255 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.541806 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.542769 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.543147 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.543197 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.543206 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:57.543340 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 09:16:57.543373 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:16:57.543497 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.541065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55037 (local address 127.0.98.2:40687)
0620 09:16:57.541109 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:57.541110 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:57.541208 (+ 98us) server_negotiation.cc:407] Connection header received
0620 09:16:57.541252 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:57.541253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:57.541266 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:57.541303 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:57.541471 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.541946 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.542821 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.543174 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.543236 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.543247 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:57.543395 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 09:16:57.543432 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:16:57.543524 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.540960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40687 (local address 127.0.98.3:55037)
0620 09:16:57.541143 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.541153 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:57.541204 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:57.541316 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:57.541317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:57.541318 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:57.541376 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.541377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.542015 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.542016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.542742 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:57.542745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.543266 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.543267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:57.543324 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.543329 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:16:57.543378 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 09:16:57.543423 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":116,"threads_started":1}
I20260620 09:16:57.543654 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f898b30117a4821b39a37c877e3c9b4" candidate_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8587525e365e445c8df92975c7dbd21c" is_pre_election: true
I20260620 09:16:57.543736 7403 raft_consensus.cc:2454] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec6b0b7db96744d3919456e1c5f5cc39 in term 0.
I20260620 09:16:57.543749 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f898b30117a4821b39a37c877e3c9b4" candidate_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc5044d7eec4e3db65214b485134ca8" is_pre_election: true
I20260620 09:16:57.543867 7535 raft_consensus.cc:2454] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec6b0b7db96744d3919456e1c5f5cc39 in term 0.
I20260620 09:16:57.543900 7611 leader_election.cc:302] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8587525e365e445c8df92975c7dbd21c, ec6b0b7db96744d3919456e1c5f5cc39; no voters:
I20260620 09:16:57.544039 7815 raft_consensus.cc:2781] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:57.544072 7815 raft_consensus.cc:482] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:57.544088 7815 raft_consensus.cc:3037] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:57.544793 7815 raft_consensus.cc:504] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } } peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } }
I20260620 09:16:57.544915 7815 leader_election.cc:288] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [CANDIDATE]: Term 1 election: Requested vote from peers 8587525e365e445c8df92975c7dbd21c (127.0.98.1:46105), fbc5044d7eec4e3db65214b485134ca8 (127.0.98.2:40687)
I20260620 09:16:57.545044 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f898b30117a4821b39a37c877e3c9b4" candidate_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8587525e365e445c8df92975c7dbd21c"
I20260620 09:16:57.545075 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f898b30117a4821b39a37c877e3c9b4" candidate_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc5044d7eec4e3db65214b485134ca8"
I20260620 09:16:57.545116 7403 raft_consensus.cc:3037] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:57.545131 7535 raft_consensus.cc:3037] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:57.545755 7403 raft_consensus.cc:2454] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6b0b7db96744d3919456e1c5f5cc39 in term 1.
I20260620 09:16:57.545851 7535 raft_consensus.cc:2454] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6b0b7db96744d3919456e1c5f5cc39 in term 1.
I20260620 09:16:57.545905 7611 leader_election.cc:302] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8587525e365e445c8df92975c7dbd21c, ec6b0b7db96744d3919456e1c5f5cc39; no voters:
I20260620 09:16:57.546015 7815 raft_consensus.cc:2781] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:57.546150 7815 raft_consensus.cc:686] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 1 LEADER]: Becoming Leader. State: Replica: ec6b0b7db96744d3919456e1c5f5cc39, State: Running, Role: LEADER
I20260620 09:16:57.546206 7815 consensus_queue.cc:237] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } } peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } }
I20260620 09:16:57.546769 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.546594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.3:37837)
0620 09:16:57.546727 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.546749 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 09:16:57.546849 7081 catalog_manager.cc:5495] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec6b0b7db96744d3919456e1c5f5cc39 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6b0b7db96744d3919456e1c5f5cc39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39167 } health_report { overall_health: HEALTHY } } }
I20260620 09:16:57.571794 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.569644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.0.98.3:39167)
0620 09:16:57.569702 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:57.569704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:57.569711 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:57.569744 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:57.569745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:57.569762 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:57.569822 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:57.569966 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.570422 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.571130 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:57.571315 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:57.571366 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:57.571475 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:57.571573 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:57.571646 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:57.571704 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 09:16:57.571722 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:16:57.572854 7403 raft_consensus.cc:1261] T 7f898b30117a4821b39a37c877e3c9b4 P 8587525e365e445c8df92975c7dbd21c [term 1 FOLLOWER]: Refusing update from remote peer ec6b0b7db96744d3919456e1c5f5cc39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:57.572904 7535 raft_consensus.cc:1261] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8 [term 1 FOLLOWER]: Refusing update from remote peer ec6b0b7db96744d3919456e1c5f5cc39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:57.573117 7815 consensus_queue.cc:1035] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8587525e365e445c8df92975c7dbd21c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:57.573262 7822 consensus_queue.cc:1035] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc5044d7eec4e3db65214b485134ca8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:57.574076 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.573895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.1:53575)
0620 09:16:57.574003 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.574034 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 09:16:57.574209 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.573962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.2:51793)
0620 09:16:57.574154 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.574182 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260620 09:16:57.574929 7828 mvcc.cc:205] Tried to move back new op lower bound from 7298854983976296448 to 7298854983869759488. Current Snapshot: MvccSnapshot[applied={T|T < 7298854983976296448}]
I20260620 09:16:57.575081 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.574906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.3:52325)
0620 09:16:57.575024 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.575056 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 09:16:57.575142 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.575012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.2:45743)
0620 09:16:57.575092 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.575117 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 09:16:57.575867 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.575727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.3:43431)
0620 09:16:57.575808 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.575831 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 09:16:57.576315 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.576232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41791 (local address 127.0.98.1:55295)
0620 09:16:57.576261 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.576288 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 09:16:57.577348 7081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:57.577422 7081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:57.578569 7081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04208839d9c74e7eb1b9db21c7cf51ea: Sending DeleteTablet for 3 replicas of tablet 7f898b30117a4821b39a37c877e3c9b4
I20260620 09:16:57.578850 7655 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f898b30117a4821b39a37c877e3c9b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:57 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36038
I20260620 09:16:57.578866 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f898b30117a4821b39a37c877e3c9b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:57 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37560
I20260620 09:16:57.578928 7519 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f898b30117a4821b39a37c877e3c9b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:57 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35442
I20260620 09:16:57.579038 7835 tablet_replica.cc:330] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39: stopping tablet replica
I20260620 09:16:57.579085 7835 raft_consensus.cc:2229] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:57.579155 7835 raft_consensus.cc:2258] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:57.579269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 7320
I20260620 09:16:57.580756 7835 ts_tablet_manager.cc:1905] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:57.581893 7837 tablet_replica.cc:330] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8: stopping tablet replica
I20260620 09:16:57.581952 7837 raft_consensus.cc:2229] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:57.581990 7837 raft_consensus.cc:2258] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:57.582316 7835 ts_tablet_manager.cc:1918] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:57.582381 7835 log.cc:1196] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/7f898b30117a4821b39a37c877e3c9b4
I20260620 09:16:57.582598 7835 ts_tablet_manager.cc:1939] T 7f898b30117a4821b39a37c877e3c9b4 P ec6b0b7db96744d3919456e1c5f5cc39: Deleting consensus metadata
I20260620 09:16:57.582876 7068 catalog_manager.cc:4841] TS ec6b0b7db96744d3919456e1c5f5cc39 (127.0.98.3:39167): tablet 7f898b30117a4821b39a37c877e3c9b4 (table test-table [id=41b2bccde9e644a49e09d16ee1b98e48]) successfully deleted
I20260620 09:16:57.585072 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.584955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46105 (local address 127.0.0.1:37570)
0620 09:16:57.585024 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.585042 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 09:16:57.585089 7837 ts_tablet_manager.cc:1905] T 7f898b30117a4821b39a37c877e3c9b4 P fbc5044d7eec4e3db65214b485134ca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:57.585199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 7456
W20260620 09:16:57.585212 7069 catalog_manager.cc:4584] TS 8587525e365e445c8df92975c7dbd21c (127.0.98.1:46105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f898b30117a4821b39a37c877e3c9b4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46105: connect: Connection refused (error 111)
I20260620 09:16:57.589672 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:57.589592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40687 (local address 127.0.0.1:35444)
0620 09:16:57.589623 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:57.589643 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 09:16:57.589983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 7592
I20260620 09:16:57.594034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 6948
I20260620 09:16:57.597790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 7136
2026-06-20T09:16:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4665 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:16:57 dist-test-slave-rq8c krb5kdc[7840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:16:57 dist-test-slave-rq8c krb5kdc[7840](info): set up 2 sockets
Jun 20 09:16:57 dist-test-slave-rq8c krb5kdc[7840](info): commencing operation
krb5kdc: starting...
W20260620 09:16:58.543835 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:16:58 dist-test-slave-rq8c krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:16:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:16:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 09:16:58.648245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33893
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:58.727496 7853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:58.727739 7853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:58.727763 7853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:58.728978 7853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:16:58.729019 7853 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 09:16:58.729032 7853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:58.729043 7853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:58.729054 7853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:16:58.729068 7853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:16:58.730540 7853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33893
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:58.730762 7853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:58.730985 7853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:58.735854 7858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:58.735944 7853 server_base.cc:1034] running on GCE node
W20260620 09:16:58.735924 7861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:58.735917 7859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:58.736399 7853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:58.736713 7853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:58.737881 7853 hybrid_clock.cc:648] HybridClock initialized: now 1781947018737861 us; error 37 us; skew 500 ppm
Jun 20 09:16:58 dist-test-slave-rq8c krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947018, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:58.740271 7853 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 09:16:58.740679 7853 webserver.cc:460] Webserver started at http://127.0.98.62:39651/ using document root <none> and password file <none>
I20260620 09:16:58.740900 7853 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:58.740945 7853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:58.741071 7853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:58.742107 7853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "2a6740cf374346938d9d6abbf1e913d8"
format_stamp: "Formatted at 2026-06-20 09:16:58 on dist-test-slave-rq8c"
server_key: "1cb578d2763bf95686136d2ae0024417"
server_key_iv: "46551394b156dee35924a1803237422d"
server_key_version: "encryptionkey@0"
I20260620 09:16:58.742430 7853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a6740cf374346938d9d6abbf1e913d8"
format_stamp: "Formatted at 2026-06-20 09:16:58 on dist-test-slave-rq8c"
server_key: "1cb578d2763bf95686136d2ae0024417"
server_key_iv: "46551394b156dee35924a1803237422d"
server_key_version: "encryptionkey@0"
I20260620 09:16:58.743631 7853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:16:58.744304 7868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:58.744457 7853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:58.744534 7853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "2a6740cf374346938d9d6abbf1e913d8"
format_stamp: "Formatted at 2026-06-20 09:16:58 on dist-test-slave-rq8c"
server_key: "1cb578d2763bf95686136d2ae0024417"
server_key_iv: "46551394b156dee35924a1803237422d"
server_key_version: "encryptionkey@0"
I20260620 09:16:58.744601 7853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:58.772826 7853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:58.774660 7853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:58.774796 7853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:58.779210 7853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33893
I20260620 09:16:58.779281 7930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33893 every 8 connection(s)
I20260620 09:16:58.779582 7853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:16:58.780288 7931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:58.782714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 7853
I20260620 09:16:58.782766 7931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8: Bootstrap starting.
I20260620 09:16:58.782805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:16:58.782987 392 external_mini_cluster.cc:1428] Setting key 369f52f85c11d37cac394700ca286e3d
I20260620 09:16:58.783385 7931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:58.783766 7931 log.cc:824] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:58.784441 7931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8: No bootstrap required, opened a new log
I20260620 09:16:58.786566 7931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6740cf374346938d9d6abbf1e913d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33893 } }
I20260620 09:16:58.786747 7931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:58.786778 7931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6740cf374346938d9d6abbf1e913d8, State: Initialized, Role: FOLLOWER
I20260620 09:16:58.786911 7931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6740cf374346938d9d6abbf1e913d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33893 } }
I20260620 09:16:58.786978 7931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:16:58.787014 7931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:16:58.787062 7931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:58.787580 7931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6740cf374346938d9d6abbf1e913d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33893 } }
I20260620 09:16:58.787715 7931 leader_election.cc:302] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a6740cf374346938d9d6abbf1e913d8; no voters:
I20260620 09:16:58.787897 7931 leader_election.cc:288] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:16:58.787941 7936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:58.788091 7936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [term 1 LEADER]: Becoming Leader. State: Replica: 2a6740cf374346938d9d6abbf1e913d8, State: Running, Role: LEADER
I20260620 09:16:58.788177 7931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:16:58.788187 7936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6740cf374346938d9d6abbf1e913d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33893 } }
Jun 20 09:16:58 dist-test-slave-rq8c krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:58.789798 7937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a6740cf374346938d9d6abbf1e913d8. Latest consensus state: current_term: 1 leader_uuid: "2a6740cf374346938d9d6abbf1e913d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6740cf374346938d9d6abbf1e913d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33893 } } }
I20260620 09:16:58.789875 7937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:58.790015 7938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a6740cf374346938d9d6abbf1e913d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6740cf374346938d9d6abbf1e913d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33893 } } }
I20260620 09:16:58.790066 7938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:16:58.790504 7942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:16:58.791384 7942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:16:58.792923 7942 catalog_manager.cc:1347] Generated new cluster ID: 0b73c16f301347eb9f53542ec9dbc231
I20260620 09:16:58.792968 7942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:16:58.793233 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:58.783482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.0.98.62:33893)
0620 09:16:58.783672 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:58.783682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:58.783710 (+ 28us) server_negotiation.cc:407] Connection header received
0620 09:16:58.784271 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:58.784279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:58.784333 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:58.784451 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:58.785227 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:58.786113 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:58.787101 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:58.787537 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:58.787648 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:58.789978 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:58.789982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:58.789983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:58.789996 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:58.791816 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:58.792120 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:58.792124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:58.792124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:58.792210 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:58.792379 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:58.792379 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:58.792380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:58.792666 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:58.792807 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:58.792914 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 09:16:58.793044 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260620 09:16:58.796792 7942 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:16:58.797474 7942 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:16:58.806447 7942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8: Generated new TSK 0
I20260620 09:16:58.806613 7942 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 09:16:58.834321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33893
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:58.919827 7957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:58.920037 7957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:58.920058 7957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:58.921188 7957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:58.921236 7957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:58.921255 7957 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:16:58.921271 7957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:16:58.922745 7957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:58.922974 7957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:58.923184 7957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:58.928663 7963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:58.928711 7962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:58.928663 7965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:58.928861 7957 server_base.cc:1034] running on GCE node
I20260620 09:16:58.929102 7957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:58.929308 7957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:58.930461 7957 hybrid_clock.cc:648] HybridClock initialized: now 1781947018930448 us; error 28 us; skew 500 ppm
Jun 20 09:16:58 dist-test-slave-rq8c krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947018, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:58.932868 7957 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 09:16:58.933260 7957 webserver.cc:460] Webserver started at http://127.0.98.1:33107/ using document root <none> and password file <none>
I20260620 09:16:58.933439 7957 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:58.933478 7957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:58.933563 7957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:58.934542 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "7f10df5b7fd74df2a3f237a89f1c176f"
format_stamp: "Formatted at 2026-06-20 09:16:58 on dist-test-slave-rq8c"
server_key: "3b2f1168777b179cba0003a5f25a4a50"
server_key_iv: "76effbcf836c48fa2180ad9e645ae3af"
server_key_version: "encryptionkey@0"
I20260620 09:16:58.934825 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7f10df5b7fd74df2a3f237a89f1c176f"
format_stamp: "Formatted at 2026-06-20 09:16:58 on dist-test-slave-rq8c"
server_key: "3b2f1168777b179cba0003a5f25a4a50"
server_key_iv: "76effbcf836c48fa2180ad9e645ae3af"
server_key_version: "encryptionkey@0"
I20260620 09:16:58.936038 7957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:58.936726 7972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:58.936874 7957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:58.936972 7957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "7f10df5b7fd74df2a3f237a89f1c176f"
format_stamp: "Formatted at 2026-06-20 09:16:58 on dist-test-slave-rq8c"
server_key: "3b2f1168777b179cba0003a5f25a4a50"
server_key_iv: "76effbcf836c48fa2180ad9e645ae3af"
server_key_version: "encryptionkey@0"
I20260620 09:16:58.937033 7957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:58.948607 7957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:58.950451 7957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:58.950577 7957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:58.951048 7957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:58.951089 7957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:58.951118 7957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:58.951143 7957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:16:58 dist-test-slave-rq8c krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947018, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:58.957854 7957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38697
I20260620 09:16:58.957926 8087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38697 every 8 connection(s)
I20260620 09:16:58.958256 7957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:16:58.959046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 7957
I20260620 09:16:58.959138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:16:58.959322 392 external_mini_cluster.cc:1428] Setting key 11053b425d513db6902a298fd870607a
I20260620 09:16:58.960412 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:58.951481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56741 (local address 127.0.98.62:33893)
0620 09:16:58.951525 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:58.951527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:58.951858 (+ 331us) server_negotiation.cc:407] Connection header received
0620 09:16:58.952339 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:58.952341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:58.952355 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:58.952438 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:58.953049 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:58.953819 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:58.954805 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:58.955111 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:58.955186 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:58.957658 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:58.957659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:58.957660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:58.957668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:58.959035 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:58.959466 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:58.959467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:58.959467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:58.959526 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:58.959786 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:58.959787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:58.959787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:58.959928 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:58.959994 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:58.960283 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 09:16:58.960349 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:16:58.960618 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:58.951507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33893 (local address 127.0.98.1:56741)
0620 09:16:58.951722 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:58.951761 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:58.952251 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:58.952489 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:58.952493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:58.952783 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:58.952976 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:58.952982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:58.953925 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:58.953926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:58.954740 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:58.954741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:58.955200 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:58.955201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:58.955327 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:58.955844 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:58.955853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:58.957572 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:58.959115 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:58.959120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:58.959123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:58.959396 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:58.959580 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:58.959581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:58.959581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:58.959728 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:58.959993 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:58.959999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:58.960157 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 09:16:58.960258 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 09:16:58.960411 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260620 09:16:58.961585 8088 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33893
I20260620 09:16:58.961663 8088 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:58.961824 8088 heartbeater.cc:511] Master 127.0.98.62:33893 requested a full tablet report, sending...
I20260620 09:16:58.962298 7885 ts_manager.cc:194] Registered new tserver with Master: 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697)
I20260620 09:16:58.963073 7885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56741
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:16:58.968364 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:58.962675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54589 (local address 127.0.98.62:33893)
0620 09:16:58.962709 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:58.962711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:58.962732 (+ 21us) server_negotiation.cc:407] Connection header received
0620 09:16:58.962773 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:58.962774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:58.962783 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:58.962827 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:58.963184 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:58.963923 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:58.964969 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:58.965375 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:58.965468 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:58.966314 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:58.966315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:58.966316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:58.966320 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:58.967542 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:58.967862 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:58.967863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:58.967863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:58.967916 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:58.968041 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:58.968042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:58.968042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:58.968124 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:58.968165 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:58.968254 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 09:16:58.968309 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:16:58.968372 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:58.962556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33893 (local address 127.0.98.1:54589)
0620 09:16:58.962677 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:58.962686 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:58.962727 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:58.962830 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:58.962830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:58.963031 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:58.963093 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:58.963094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:58.964033 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:58.964033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:58.964862 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:58.964865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:58.965470 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:58.965471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:58.965538 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:58.965895 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:58.965900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:58.966243 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:58.967610 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:58.967610 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:58.967611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:58.967793 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:58.967963 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:58.967963 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:58.967964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:58.968008 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:58.968161 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:58.968162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:58.968195 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 09:16:58.968238 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 09:16:58.968305 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 09:16:58.980073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33893
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:59.056869 8097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:59.057080 8097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:59.057101 8097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:59.058220 8097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:59.058259 8097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:59.058279 8097 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:16:59.058296 8097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:16:59.059751 8097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:59.059957 8097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:59.060156 8097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:59.064780 8105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:59.064826 8097 server_base.cc:1034] running on GCE node
W20260620 09:16:59.064750 8103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:59.064750 8102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:59.065167 8097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:59.065375 8097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:59.066504 8097 hybrid_clock.cc:648] HybridClock initialized: now 1781947019066494 us; error 28 us; skew 500 ppm
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:59.068994 8097 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 09:16:59.069382 8097 webserver.cc:460] Webserver started at http://127.0.98.2:36443/ using document root <none> and password file <none>
I20260620 09:16:59.069577 8097 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:59.069620 8097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:59.069741 8097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:59.070709 8097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89"
format_stamp: "Formatted at 2026-06-20 09:16:59 on dist-test-slave-rq8c"
server_key: "19862bd802186e8c8e0cc44d5b45bb6e"
server_key_iv: "165702a4c39205942873538325ff5869"
server_key_version: "encryptionkey@0"
I20260620 09:16:59.071017 8097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89"
format_stamp: "Formatted at 2026-06-20 09:16:59 on dist-test-slave-rq8c"
server_key: "19862bd802186e8c8e0cc44d5b45bb6e"
server_key_iv: "165702a4c39205942873538325ff5869"
server_key_version: "encryptionkey@0"
I20260620 09:16:59.072237 8097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:59.072896 8112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:59.073037 8097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:59.073112 8097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89"
format_stamp: "Formatted at 2026-06-20 09:16:59 on dist-test-slave-rq8c"
server_key: "19862bd802186e8c8e0cc44d5b45bb6e"
server_key_iv: "165702a4c39205942873538325ff5869"
server_key_version: "encryptionkey@0"
I20260620 09:16:59.073176 8097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:59.093120 8097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:59.095582 8097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:59.095733 8097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:59.096259 8097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:59.096294 8097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:59.096320 8097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:59.096336 8097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:59.103858 8097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39735
I20260620 09:16:59.104300 8097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:16:59.104827 8227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39735 every 8 connection(s)
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:59.109678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 8097
I20260620 09:16:59.109788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:16:59.109997 392 external_mini_cluster.cc:1428] Setting key 33ac01f2283244a6a426ee67716f9144
I20260620 09:16:59.110075 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.100157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49173 (local address 127.0.98.62:33893)
0620 09:16:59.100205 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.100206 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.101279 (+ 1073us) server_negotiation.cc:407] Connection header received
0620 09:16:59.101937 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.101939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.101964 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.102010 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:59.103266 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.104013 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.105553 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.105758 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.105816 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.107847 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:59.107849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:59.107849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:59.107856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:59.108940 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.109272 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.109273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.109273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.109316 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.109491 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.109491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.109492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.109608 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:59.109674 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.109916 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.109987 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:16:59.110318 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.100121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33893 (local address 127.0.98.2:49173)
0620 09:16:59.101130 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.101177 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.101842 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.102098 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.102102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.102459 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:59.103132 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.103139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.104180 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.104181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.105479 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.105482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.105856 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.105858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.105976 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.106458 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:59.106465 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:59.107779 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:59.109004 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:59.109009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:59.109011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:59.109203 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:59.109360 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:59.109360 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:59.109360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:59.109451 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:59.109661 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:59.109664 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:59.109781 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.109881 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.109996 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"thread_start_us":850,"threads_started":1}
I20260620 09:16:59.110800 8228 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33893
I20260620 09:16:59.110891 8228 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:59.111060 8228 heartbeater.cc:511] Master 127.0.98.62:33893 requested a full tablet report, sending...
I20260620 09:16:59.111670 7884 ts_manager.cc:194] Registered new tserver with Master: 6a28a3d6afac4f1b9aa5fbf2494aaf89 (127.0.98.2:39735)
I20260620 09:16:59.112085 7884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49173
I20260620 09:16:59.117008 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.112061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49437 (local address 127.0.98.62:33893)
0620 09:16:59.112106 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.112107 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.112114 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:59.112131 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.112131 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.112139 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.112189 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:59.112553 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.113131 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.113773 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.114017 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.114073 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.114880 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:59.114881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:59.114882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:59.114888 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:59.116047 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.116327 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.116328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.116329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.116377 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.116527 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.116528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.116528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.116656 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:59.116710 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.116838 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.116905 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:16:59.117020 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.111711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33893 (local address 127.0.98.2:49437)
0620 09:16:59.111847 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.111858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.111909 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.112221 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.112222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.112381 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:59.112465 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.112467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.113217 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.113217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.113699 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.113700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.114081 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.114082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.114142 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.114562 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:59.114565 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:59.114810 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:59.116114 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:59.116115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:59.116115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:59.116272 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:59.116427 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:59.116427 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:59.116428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:59.116481 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:59.116709 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:59.116710 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:59.116751 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.116810 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.116904 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 09:16:59.132313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33893
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:16:59.211258 8237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:16:59.211478 8237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:16:59.211506 8237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:16:59.212670 8237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:16:59.212708 8237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:16:59.212728 8237 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:16:59.212743 8237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:16:59.214177 8237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:16:59.214391 8237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:16:59.214604 8237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:16:59.219242 8245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:59.219223 8242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:16:59.219280 8243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:16:59.219406 8237 server_base.cc:1034] running on GCE node
I20260620 09:16:59.219663 8237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:16:59.219904 8237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:16:59.221071 8237 hybrid_clock.cc:648] HybridClock initialized: now 1781947019221058 us; error 29 us; skew 500 ppm
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:16:59.223438 8237 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 09:16:59.223875 8237 webserver.cc:460] Webserver started at http://127.0.98.3:33057/ using document root <none> and password file <none>
I20260620 09:16:59.224080 8237 fs_manager.cc:359] Metadata directory not provided
I20260620 09:16:59.224125 8237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:16:59.224241 8237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:16:59.225219 8237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "9a65b02a291247b4ac5b5ad2477ace63"
format_stamp: "Formatted at 2026-06-20 09:16:59 on dist-test-slave-rq8c"
server_key: "786496f30f41c944302c30803e051672"
server_key_iv: "ab00b36d8fc2a73790db41069d9c11f7"
server_key_version: "encryptionkey@0"
I20260620 09:16:59.225533 8237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9a65b02a291247b4ac5b5ad2477ace63"
format_stamp: "Formatted at 2026-06-20 09:16:59 on dist-test-slave-rq8c"
server_key: "786496f30f41c944302c30803e051672"
server_key_iv: "ab00b36d8fc2a73790db41069d9c11f7"
server_key_version: "encryptionkey@0"
I20260620 09:16:59.226778 8237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:16:59.227421 8252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:59.227576 8237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:59.227675 8237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "9a65b02a291247b4ac5b5ad2477ace63"
format_stamp: "Formatted at 2026-06-20 09:16:59 on dist-test-slave-rq8c"
server_key: "786496f30f41c944302c30803e051672"
server_key_iv: "ab00b36d8fc2a73790db41069d9c11f7"
server_key_version: "encryptionkey@0"
I20260620 09:16:59.227732 8237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:16:59.248883 8237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:16:59.250613 8237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:16:59.250741 8237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:16:59.251215 8237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:16:59.251256 8237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:59.251286 8237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:16:59.251303 8237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:59.257666 8237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46747
I20260620 09:16:59.257730 8367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46747 every 8 connection(s)
I20260620 09:16:59.258008 8237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:16:59.260273 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.251642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59837 (local address 127.0.98.62:33893)
0620 09:16:59.251694 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.251696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.251864 (+ 168us) server_negotiation.cc:407] Connection header received
0620 09:16:59.252343 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.252344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.252356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.252436 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:59.253197 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.253947 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.254921 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.255209 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.255282 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.257787 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:59.257789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:59.257790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:59.257797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:59.259139 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.259498 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.259498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.259499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.259542 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.259773 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.259774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.259774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.259883 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:59.259930 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.260154 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.260210 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:16:59.260463 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.251590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33893 (local address 127.0.98.3:59837)
0620 09:16:59.251747 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.251786 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.252260 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.252501 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.252505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.252806 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:59.253089 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.253096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.254038 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.254039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.254855 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.254858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.255294 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.255295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.255442 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.255951 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:59.255960 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:59.257683 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:59.259215 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:59.259220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:59.259222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:59.259440 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:59.259580 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:59.259580 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:59.259580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:59.259716 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:59.259934 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:59.259939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:59.260051 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.260128 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.260246 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260620 09:16:59.260898 8368 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33893
I20260620 09:16:59.260989 8368 heartbeater.cc:464] Registering TS with master...
I20260620 09:16:59.261144 8368 heartbeater.cc:511] Master 127.0.98.62:33893 requested a full tablet report, sending...
I20260620 09:16:59.261495 7885 ts_manager.cc:194] Registered new tserver with Master: 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747)
I20260620 09:16:59.261966 7885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59837
I20260620 09:16:59.266736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 8237
I20260620 09:16:59.266827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:16:59.267009 392 external_mini_cluster.cc:1428] Setting key 524ebcd9256be36e1a061aaa142f3c58
I20260620 09:16:59.267956 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:16:59.269344 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.261859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56817 (local address 127.0.98.62:33893)
0620 09:16:59.261892 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.261893 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.261959 (+ 66us) server_negotiation.cc:407] Connection header received
0620 09:16:59.262006 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.262006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.262016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.262071 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:59.262478 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.262985 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.263686 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.263882 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.263934 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.267152 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:59.267154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:59.267154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:59.267161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:59.268418 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.268699 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.268700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.268700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.268748 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.268902 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.268903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.268904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.269002 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:59.269055 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.269182 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.269246 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:16:59.269356 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.261797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33893 (local address 127.0.98.3:56817)
0620 09:16:59.261908 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.261918 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.261969 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.262160 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.262161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.262353 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:59.262406 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.262408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.263074 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.263075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.263590 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.263591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.263937 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.263938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.264003 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.266756 (+ 2753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:59.266761 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:59.267086 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:59.268486 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:59.268487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:59.268487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:59.268642 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:59.268802 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:59.268802 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:59.268803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:59.268855 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:59.269051 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:59.269052 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:59.269092 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.269153 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.269247 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":62,"threads_started":1}
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:16:59.287413 7934 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:16:59.282488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.0.98.62:33893)
0620 09:16:59.282547 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.282549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.282559 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:16:59.282586 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.282587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.282597 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.282640 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:59.283150 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.283690 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.284423 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.284611 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.284660 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.287318 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49496: BlockingRecv error: recv got EOF from 127.0.0.1:49496 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:59.293835 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.288096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.0.98.62:33893)
0620 09:16:59.288143 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.288145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.288227 (+ 82us) server_negotiation.cc:407] Connection header received
0620 09:16:59.288274 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.288275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.288282 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.288324 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:59.288703 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.289180 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.289845 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.290046 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.290109 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.291952 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:59.291953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:59.291953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:59.291959 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:59.292967 (+ 1008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.293317 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.293318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.293318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.293368 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.293502 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.293503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.293503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.293590 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:59.293656 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.293723 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.293774 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:16:59.295176 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:59.296705 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:59.302487 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.298554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38697 (local address 127.0.0.1:44292)
0620 09:16:59.299041 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.299048 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.299089 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.299226 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.299227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.299228 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:59.299265 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.299266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.300553 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.300555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.301469 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.301471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.302204 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.302205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.302279 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.302293 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.302367 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.302416 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":30,"thread_start_us":48,"threads_started":1}
I20260620 09:16:59.302524 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.298885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44292 (local address 127.0.98.1:38697)
0620 09:16:59.299036 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.299039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.299088 (+ 49us) server_negotiation.cc:407] Connection header received
0620 09:16:59.299140 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.299142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.299156 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.299225 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:59.299398 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.300476 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.301525 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.302125 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.302195 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.302266 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.302388 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.302425 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260620 09:16:59.302803 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.298668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.0.98.3:46747)
0620 09:16:59.298972 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.298976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.298987 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:16:59.299025 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.299027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.299057 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.299142 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:59.299295 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.300447 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.301828 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.302381 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.302432 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.302499 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.302676 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.302715 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":92,"threads_started":1}
I20260620 09:16:59.302933 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.298719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39735 (local address 127.0.0.1:52588)
0620 09:16:59.299224 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.299234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.299315 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.299556 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.299557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.299558 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:59.299613 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.299641 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.300434 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.300435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.301725 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.301727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.302666 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.302667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.302736 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.302740 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.302817 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.302866 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":68,"thread_start_us":40,"threads_started":1}
I20260620 09:16:59.302983 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.298800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.0.98.2:39735)
0620 09:16:59.298937 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.298942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.299288 (+ 346us) server_negotiation.cc:407] Connection header received
0620 09:16:59.299402 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.299404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.299419 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.299470 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:59.299691 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.300362 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.301813 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.302502 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.302563 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.302632 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.302845 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.302890 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260620 09:16:59.302753 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.298541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46747 (local address 127.0.0.1:53476)
0620 09:16:59.298815 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.298854 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.298914 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.299152 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.299155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.299156 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:59.299213 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.299215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.300513 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.300514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.301740 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.301741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.302532 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.302533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.302605 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.302609 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.302659 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.302690 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":44,"thread_start_us":49,"threads_started":1}
I20260620 09:16:59.303465 8022 tablet_service.cc:1467] Processing CreateTablet for tablet 5252a7fae6564cb187b78f651458cc65 (DEFAULT_TABLE table=test-table [id=8fae72a38f564c058bdd5882c5470b86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:59.303637 8302 tablet_service.cc:1467] Processing CreateTablet for tablet 5252a7fae6564cb187b78f651458cc65 (DEFAULT_TABLE table=test-table [id=8fae72a38f564c058bdd5882c5470b86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:59.303637 8155 tablet_service.cc:1467] Processing CreateTablet for tablet 5252a7fae6564cb187b78f651458cc65 (DEFAULT_TABLE table=test-table [id=8fae72a38f564c058bdd5882c5470b86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:59.303845 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5252a7fae6564cb187b78f651458cc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.303956 8155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5252a7fae6564cb187b78f651458cc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.303956 8302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5252a7fae6564cb187b78f651458cc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.306679 8393 tablet_bootstrap.cc:492] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f: Bootstrap starting.
I20260620 09:16:59.307248 8393 tablet_bootstrap.cc:654] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.307427 8394 tablet_bootstrap.cc:492] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63: Bootstrap starting.
I20260620 09:16:59.307601 8393 log.cc:824] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:59.307997 8394 tablet_bootstrap.cc:654] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.308284 8394 log.cc:824] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:59.308348 8393 tablet_bootstrap.cc:492] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f: No bootstrap required, opened a new log
I20260620 09:16:59.308408 8393 ts_tablet_manager.cc:1397] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:16:59.308598 8395 tablet_bootstrap.cc:492] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Bootstrap starting.
I20260620 09:16:59.308923 8394 tablet_bootstrap.cc:492] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63: No bootstrap required, opened a new log
I20260620 09:16:59.308980 8394 ts_tablet_manager.cc:1397] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:59.309118 8395 tablet_bootstrap.cc:654] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.309422 8395 log.cc:824] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Log is configured to *not* fsync() on all Append() calls
I20260620 09:16:59.310166 8395 tablet_bootstrap.cc:492] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: No bootstrap required, opened a new log
I20260620 09:16:59.310233 8395 ts_tablet_manager.cc:1397] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:59.310321 8393 raft_consensus.cc:348] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } }
I20260620 09:16:59.310477 8393 raft_consensus.cc:374] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.310515 8393 raft_consensus.cc:729] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f10df5b7fd74df2a3f237a89f1c176f, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.310619 8393 consensus_queue.cc:260] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } }
I20260620 09:16:59.310652 8394 raft_consensus.cc:348] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } }
I20260620 09:16:59.310810 8394 raft_consensus.cc:374] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.310853 8394 raft_consensus.cc:729] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a65b02a291247b4ac5b5ad2477ace63, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.310876 8393 ts_tablet_manager.cc:1428] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 09:16:59.310959 8394 consensus_queue.cc:260] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } }
I20260620 09:16:59.311003 8088 heartbeater.cc:503] Master 127.0.98.62:33893 was elected leader, sending a full tablet report...
I20260620 09:16:59.311203 8394 ts_tablet_manager.cc:1428] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:16:59.311247 8368 heartbeater.cc:503] Master 127.0.98.62:33893 was elected leader, sending a full tablet report...
I20260620 09:16:59.312238 8395 raft_consensus.cc:348] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } }
I20260620 09:16:59.312357 8395 raft_consensus.cc:374] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.312383 8395 raft_consensus.cc:729] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a28a3d6afac4f1b9aa5fbf2494aaf89, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.312463 8395 consensus_queue.cc:260] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } }
I20260620 09:16:59.312666 8395 ts_tablet_manager.cc:1428] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:16:59.312693 8228 heartbeater.cc:503] Master 127.0.98.62:33893 was elected leader, sending a full tablet report...
W20260620 09:16:59.354890 8229 tablet.cc:2369] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:16:59.459151 8089 tablet.cc:2369] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:16:59.501724 8400 raft_consensus.cc:482] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:59.501871 8400 raft_consensus.cc:504] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } }
I20260620 09:16:59.502238 8400 leader_election.cc:288] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697), 6a28a3d6afac4f1b9aa5fbf2494aaf89 (127.0.98.2:39735)
I20260620 09:16:59.506085 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.502673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58201 (local address 127.0.98.2:39735)
0620 09:16:59.502865 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.502867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.502874 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:59.502894 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.502895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.502906 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.502953 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:59.503077 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.503771 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.505128 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.505713 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.505791 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.505799 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.505973 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.506011 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260620 09:16:59.506095 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.502520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53861 (local address 127.0.98.1:38697)
0620 09:16:59.502582 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.502584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.502594 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:16:59.502638 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.502639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.502650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.502695 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:59.502881 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.503664 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.505128 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.505707 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.505787 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.505799 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.505974 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.506014 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:16:59.506106 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.502551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39735 (local address 127.0.98.3:58201)
0620 09:16:59.502739 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.502758 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.502793 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.502966 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.502967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.502968 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:59.503017 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.503018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.503828 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.503829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.505020 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.505022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.505801 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.505801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.505903 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.505910 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.505960 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.506004 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":34,"thread_start_us":119,"threads_started":1}
I20260620 09:16:59.506116 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.502441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38697 (local address 127.0.98.3:53861)
0620 09:16:59.502502 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.502513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.502579 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.502736 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.502738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.502740 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:59.502804 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.502805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.503751 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.503752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.505020 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.505022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.505800 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.505801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.505903 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.505910 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.505960 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.506004 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":15}
I20260620 09:16:59.506479 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5252a7fae6564cb187b78f651458cc65" candidate_uuid: "9a65b02a291247b4ac5b5ad2477ace63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" is_pre_election: true
I20260620 09:16:59.506649 8173 raft_consensus.cc:2454] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a65b02a291247b4ac5b5ad2477ace63 in term 0.
I20260620 09:16:59.506632 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5252a7fae6564cb187b78f651458cc65" candidate_uuid: "9a65b02a291247b4ac5b5ad2477ace63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" is_pre_election: true
I20260620 09:16:59.506790 8042 raft_consensus.cc:2454] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a65b02a291247b4ac5b5ad2477ace63 in term 0.
I20260620 09:16:59.507027 8253 leader_election.cc:302] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a28a3d6afac4f1b9aa5fbf2494aaf89, 9a65b02a291247b4ac5b5ad2477ace63; no voters:
I20260620 09:16:59.507144 8400 raft_consensus.cc:2781] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:59.507198 8400 raft_consensus.cc:482] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:59.507228 8400 raft_consensus.cc:3037] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:59.508220 8400 raft_consensus.cc:504] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } }
I20260620 09:16:59.508327 8400 leader_election.cc:288] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [CANDIDATE]: Term 1 election: Requested vote from peers 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697), 6a28a3d6afac4f1b9aa5fbf2494aaf89 (127.0.98.2:39735)
I20260620 09:16:59.508484 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5252a7fae6564cb187b78f651458cc65" candidate_uuid: "9a65b02a291247b4ac5b5ad2477ace63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f10df5b7fd74df2a3f237a89f1c176f"
W20260620 09:16:59.508535 8369 tablet.cc:2369] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:16:59.508520 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5252a7fae6564cb187b78f651458cc65" candidate_uuid: "9a65b02a291247b4ac5b5ad2477ace63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89"
I20260620 09:16:59.508589 8173 raft_consensus.cc:3037] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:59.508586 8042 raft_consensus.cc:3037] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:59.509225 8173 raft_consensus.cc:2454] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a65b02a291247b4ac5b5ad2477ace63 in term 1.
I20260620 09:16:59.509264 8042 raft_consensus.cc:2454] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a65b02a291247b4ac5b5ad2477ace63 in term 1.
I20260620 09:16:59.509393 8253 leader_election.cc:302] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f10df5b7fd74df2a3f237a89f1c176f, 9a65b02a291247b4ac5b5ad2477ace63; no voters:
I20260620 09:16:59.509492 8400 raft_consensus.cc:2781] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:59.509629 8400 raft_consensus.cc:686] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 LEADER]: Becoming Leader. State: Replica: 9a65b02a291247b4ac5b5ad2477ace63, State: Running, Role: LEADER
I20260620 09:16:59.509718 8400 consensus_queue.cc:237] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } }
I20260620 09:16:59.510465 7885 catalog_manager.cc:5495] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9a65b02a291247b4ac5b5ad2477ace63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } health_report { overall_health: UNKNOWN } } }
I20260620 09:16:59.526808 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.524540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53492 (local address 127.0.98.3:46747)
0620 09:16:59.524584 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.524586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.524594 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:16:59.524661 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.524662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.524681 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.524737 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:59.524863 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.525352 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.526123 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.526310 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.526362 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.526514 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:59.526610 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:59.526676 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.526710 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.526729 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:16:59.528340 8042 raft_consensus.cc:1261] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Refusing update from remote peer 9a65b02a291247b4ac5b5ad2477ace63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:59.528368 8173 raft_consensus.cc:1261] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Refusing update from remote peer 9a65b02a291247b4ac5b5ad2477ace63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:59.528661 8403 consensus_queue.cc:1035] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:59.528788 8400 consensus_queue.cc:1035] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:59.531158 8408 mvcc.cc:205] Tried to move back new op lower bound from 7298854991985864704 to 7298854991912456192. Current Snapshot: MvccSnapshot[applied={T|T < 7298854991985864704}]
I20260620 09:16:59.533507 7885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:59.533579 7885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:59.534215 7885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8: Sending DeleteTablet for 3 replicas of tablet 5252a7fae6564cb187b78f651458cc65
I20260620 09:16:59.534559 8022 tablet_service.cc:1514] Processing DeleteTablet for tablet 5252a7fae6564cb187b78f651458cc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44292
I20260620 09:16:59.534575 8155 tablet_service.cc:1514] Processing DeleteTablet for tablet 5252a7fae6564cb187b78f651458cc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52588
I20260620 09:16:59.534612 8302 tablet_service.cc:1514] Processing DeleteTablet for tablet 5252a7fae6564cb187b78f651458cc65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53476
I20260620 09:16:59.534966 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:59.535075 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:59.535706 8415 tablet_replica.cc:330] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f: stopping tablet replica
I20260620 09:16:59.535858 8417 tablet_replica.cc:330] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63: stopping tablet replica
I20260620 09:16:59.535952 8417 raft_consensus.cc:2229] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:59.536065 8417 raft_consensus.cc:2258] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:59.536103 8022 tablet_service.cc:1467] Processing CreateTablet for tablet 55bc08f2f3f14092bbb259214a8279a4 (DEFAULT_TABLE table=test-table [id=8fc5722dbfcf4bbdb6b16f89b74e59fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:59.536242 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55bc08f2f3f14092bbb259214a8279a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.536495 8155 tablet_service.cc:1467] Processing CreateTablet for tablet 55bc08f2f3f14092bbb259214a8279a4 (DEFAULT_TABLE table=test-table [id=8fc5722dbfcf4bbdb6b16f89b74e59fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:59.536586 8155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55bc08f2f3f14092bbb259214a8279a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.536580 8417 ts_tablet_manager.cc:1905] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:59.537602 8393 tablet_bootstrap.cc:492] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Bootstrap starting.
I20260620 09:16:59.536226 8415 raft_consensus.cc:2229] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:59.537742 8415 raft_consensus.cc:2258] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:59.537866 8395 tablet_bootstrap.cc:492] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Bootstrap starting.
I20260620 09:16:59.537895 8416 tablet_replica.cc:330] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: stopping tablet replica
I20260620 09:16:59.537957 8393 tablet_bootstrap.cc:654] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.538000 8416 raft_consensus.cc:2229] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:59.538095 8416 raft_consensus.cc:2258] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:59.538228 8395 tablet_bootstrap.cc:654] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.538465 8415 ts_tablet_manager.cc:1905] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:59.538789 8417 ts_tablet_manager.cc:1918] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:59.538851 8417 log.cc:1196] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/5252a7fae6564cb187b78f651458cc65
I20260620 09:16:59.538854 8302 tablet_service.cc:1467] Processing CreateTablet for tablet 55bc08f2f3f14092bbb259214a8279a4 (DEFAULT_TABLE table=test-table [id=8fc5722dbfcf4bbdb6b16f89b74e59fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:59.539008 8302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55bc08f2f3f14092bbb259214a8279a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.539079 8417 ts_tablet_manager.cc:1939] T 5252a7fae6564cb187b78f651458cc65 P 9a65b02a291247b4ac5b5ad2477ace63: Deleting consensus metadata
I20260620 09:16:59.539901 8415 ts_tablet_manager.cc:1918] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:59.539958 8415 log.cc:1196] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/wal/wals/5252a7fae6564cb187b78f651458cc65
I20260620 09:16:59.539947 8416 ts_tablet_manager.cc:1905] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:59.540048 8394 tablet_bootstrap.cc:492] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63: Bootstrap starting.
I20260620 09:16:59.540176 8415 ts_tablet_manager.cc:1939] T 5252a7fae6564cb187b78f651458cc65 P 7f10df5b7fd74df2a3f237a89f1c176f: Deleting consensus metadata
I20260620 09:16:59.540412 8394 tablet_bootstrap.cc:654] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.540830 7870 catalog_manager.cc:4841] TS 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747): tablet 5252a7fae6564cb187b78f651458cc65 (table test-table [id=8fae72a38f564c058bdd5882c5470b86]) successfully deleted
I20260620 09:16:59.540956 7869 catalog_manager.cc:4841] TS 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697): tablet 5252a7fae6564cb187b78f651458cc65 (table test-table [id=8fae72a38f564c058bdd5882c5470b86]) successfully deleted
I20260620 09:16:59.541296 8416 ts_tablet_manager.cc:1918] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:59.541354 8416 log.cc:1196] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/5252a7fae6564cb187b78f651458cc65
I20260620 09:16:59.541451 8394 tablet_bootstrap.cc:492] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63: No bootstrap required, opened a new log
I20260620 09:16:59.541484 8394 ts_tablet_manager.cc:1397] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:59.541563 8416 ts_tablet_manager.cc:1939] T 5252a7fae6564cb187b78f651458cc65 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Deleting consensus metadata
I20260620 09:16:59.541610 8394 raft_consensus.cc:348] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.541668 8394 raft_consensus.cc:374] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.541687 8394 raft_consensus.cc:729] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a65b02a291247b4ac5b5ad2477ace63, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.541723 8394 consensus_queue.cc:260] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.541805 8394 ts_tablet_manager.cc:1428] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:59.541980 7869 catalog_manager.cc:4841] TS 6a28a3d6afac4f1b9aa5fbf2494aaf89 (127.0.98.2:39735): tablet 5252a7fae6564cb187b78f651458cc65 (table test-table [id=8fae72a38f564c058bdd5882c5470b86]) successfully deleted
I20260620 09:16:59.542188 8395 tablet_bootstrap.cc:492] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: No bootstrap required, opened a new log
I20260620 09:16:59.542236 8395 ts_tablet_manager.cc:1397] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 09:16:59.542379 8395 raft_consensus.cc:348] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.542443 8395 raft_consensus.cc:374] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.542462 8395 raft_consensus.cc:729] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a28a3d6afac4f1b9aa5fbf2494aaf89, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.542502 8395 consensus_queue.cc:260] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.542583 8395 ts_tablet_manager.cc:1428] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:59.544009 8393 tablet_bootstrap.cc:492] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: No bootstrap required, opened a new log
I20260620 09:16:59.544075 8393 ts_tablet_manager.cc:1397] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 09:16:59.544224 8393 raft_consensus.cc:348] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.544306 8393 raft_consensus.cc:374] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.544329 8393 raft_consensus.cc:729] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f10df5b7fd74df2a3f237a89f1c176f, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.544379 8393 consensus_queue.cc:260] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.544504 8393 ts_tablet_manager.cc:1428] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:59.758271 8401 raft_consensus.cc:482] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:59.758373 8401 raft_consensus.cc:504] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.758674 8401 leader_election.cc:288] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697), 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747)
I20260620 09:16:59.761516 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.758951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43125 (local address 127.0.98.3:46747)
0620 09:16:59.758996 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.758997 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.759105 (+ 108us) server_negotiation.cc:407] Connection header received
0620 09:16:59.759159 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.759160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.759168 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.759220 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:59.759348 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.759870 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.760822 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.761204 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.761271 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.761282 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.761409 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.761443 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:16:59.761533 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.758878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46747 (local address 127.0.98.2:43125)
0620 09:16:59.759028 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.759044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.759103 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.759222 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.759223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.759225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:59.759292 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.759294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.759942 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.759943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.760752 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.760754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.761285 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.761286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.761352 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.761355 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.761399 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.761436 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":19,"thread_start_us":57,"threads_started":1}
I20260620 09:16:59.761571 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.758989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41219 (local address 127.0.98.1:38697)
0620 09:16:59.759028 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.759030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.759079 (+ 49us) server_negotiation.cc:407] Connection header received
0620 09:16:59.759145 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.759146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.759154 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.759206 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:16:59.759336 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.759843 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.760839 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.761283 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.761342 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.761351 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.761459 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.761491 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:16:59.761601 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.758877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38697 (local address 127.0.98.2:41219)
0620 09:16:59.759018 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.759029 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.759093 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.759213 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.759214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.759216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:16:59.759280 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.759282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.759909 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.759910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.760761 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.760763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.761342 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.761343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.761404 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.761407 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.761450 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.761491 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260620 09:16:59.761837 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55bc08f2f3f14092bbb259214a8279a4" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" is_pre_election: true
I20260620 09:16:59.761922 8042 raft_consensus.cc:2454] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 0.
I20260620 09:16:59.761956 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55bc08f2f3f14092bbb259214a8279a4" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a65b02a291247b4ac5b5ad2477ace63" is_pre_election: true
I20260620 09:16:59.762092 8322 raft_consensus.cc:2454] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 0.
I20260620 09:16:59.762097 8113 leader_election.cc:302] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a28a3d6afac4f1b9aa5fbf2494aaf89, 7f10df5b7fd74df2a3f237a89f1c176f; no voters:
I20260620 09:16:59.762228 8401 raft_consensus.cc:2781] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:59.762269 8401 raft_consensus.cc:482] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:59.762283 8401 raft_consensus.cc:3037] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:59.762921 8401 raft_consensus.cc:504] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.763034 8401 leader_election.cc:288] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 election: Requested vote from peers 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697), 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747)
I20260620 09:16:59.763151 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55bc08f2f3f14092bbb259214a8279a4" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f10df5b7fd74df2a3f237a89f1c176f"
I20260620 09:16:59.763168 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55bc08f2f3f14092bbb259214a8279a4" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a65b02a291247b4ac5b5ad2477ace63"
I20260620 09:16:59.763231 8322 raft_consensus.cc:3037] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:59.763214 8042 raft_consensus.cc:3037] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:59.763875 8322 raft_consensus.cc:2454] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 1.
I20260620 09:16:59.763887 8042 raft_consensus.cc:2454] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 1.
I20260620 09:16:59.764006 8113 leader_election.cc:302] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a28a3d6afac4f1b9aa5fbf2494aaf89, 7f10df5b7fd74df2a3f237a89f1c176f; no voters:
I20260620 09:16:59.764099 8401 raft_consensus.cc:2781] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:59.764192 8401 raft_consensus.cc:686] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 LEADER]: Becoming Leader. State: Replica: 6a28a3d6afac4f1b9aa5fbf2494aaf89, State: Running, Role: LEADER
I20260620 09:16:59.764269 8401 consensus_queue.cc:237] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.764874 7885 catalog_manager.cc:5495] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a28a3d6afac4f1b9aa5fbf2494aaf89 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } health_report { overall_health: UNKNOWN } } }
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947018, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:59.798869 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.792286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33893 (local address 127.0.0.1:49512)
0620 09:16:59.792463 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.792472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.792526 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.792657 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.792657 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.792941 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:16:59.792995 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.792997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.793690 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.793692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.794260 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.794261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.794632 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.794633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.794682 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.795171 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:16:59.795176 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:16:59.796549 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:16:59.797997 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:59.798004 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:59.798005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:59.798196 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:59.798339 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:16:59.798340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:16:59.798340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:16:59.798389 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:16:59.798585 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:16:59.798590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:16:59.798637 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.798700 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.798791 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
I20260620 09:16:59.798933 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.792349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49512 (local address 127.0.98.62:33893)
0620 09:16:59.792400 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.792402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.792522 (+ 120us) server_negotiation.cc:407] Connection header received
0620 09:16:59.792569 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.792570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.792579 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.792646 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:59.793085 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.793600 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.794355 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.794542 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.794615 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.796638 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:59.796639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:59.796639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:59.796645 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:59.797926 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.798251 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.798252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.798252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.798296 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.798429 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.798429 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.798429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.798518 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:59.798564 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.798758 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.798843 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":29}
I20260620 09:16:59.800755 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49512:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 09:16:59.800936 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:59.802160 8302 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6d932f2f084f89915100bc9cf066a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c75da4f923e94791a3deb9897a039440]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:16:59.802292 8302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6d932f2f084f89915100bc9cf066a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.802398 8022 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6d932f2f084f89915100bc9cf066a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c75da4f923e94791a3deb9897a039440]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:16:59.802526 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6d932f2f084f89915100bc9cf066a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.803481 8394 tablet_bootstrap.cc:492] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63: Bootstrap starting.
I20260620 09:16:59.803555 8155 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6d932f2f084f89915100bc9cf066a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c75da4f923e94791a3deb9897a039440]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:16:59.803711 8155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6d932f2f084f89915100bc9cf066a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.803807 8393 tablet_bootstrap.cc:492] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Bootstrap starting.
I20260620 09:16:59.803890 8394 tablet_bootstrap.cc:654] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.804190 8393 tablet_bootstrap.cc:654] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.804533 8394 tablet_bootstrap.cc:492] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63: No bootstrap required, opened a new log
I20260620 09:16:59.804651 8394 ts_tablet_manager.cc:1397] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:59.804792 8394 raft_consensus.cc:348] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.804875 8394 raft_consensus.cc:374] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.804896 8394 raft_consensus.cc:729] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a65b02a291247b4ac5b5ad2477ace63, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.804911 8393 tablet_bootstrap.cc:492] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f: No bootstrap required, opened a new log
I20260620 09:16:59.804948 8393 ts_tablet_manager.cc:1397] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:16:59.804955 8394 consensus_queue.cc:260] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.805043 8394 ts_tablet_manager.cc:1428] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:59.805073 8393 raft_consensus.cc:348] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.805130 8393 raft_consensus.cc:374] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.805145 8393 raft_consensus.cc:729] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f10df5b7fd74df2a3f237a89f1c176f, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.805146 8403 tablet_replica.cc:439] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } }
I20260620 09:16:59.805183 8393 consensus_queue.cc:260] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.805205 8403 tablet_replica.cc:442] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:59.805279 8393 ts_tablet_manager.cc:1428] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:59.805274 8395 tablet_bootstrap.cc:492] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Bootstrap starting.
I20260620 09:16:59.805642 8395 tablet_bootstrap.cc:654] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.806066 8399 tablet_replica.cc:439] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } }
I20260620 09:16:59.806166 8399 tablet_replica.cc:442] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:59.806362 8395 tablet_bootstrap.cc:492] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: No bootstrap required, opened a new log
I20260620 09:16:59.806411 8395 ts_tablet_manager.cc:1397] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:59.806542 8395 raft_consensus.cc:348] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.806595 8395 raft_consensus.cc:374] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.806617 8395 raft_consensus.cc:729] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a28a3d6afac4f1b9aa5fbf2494aaf89, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.806663 8395 consensus_queue.cc:260] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.806761 8395 ts_tablet_manager.cc:1428] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:59.806803 8401 tablet_replica.cc:439] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } }
I20260620 09:16:59.806874 8401 tablet_replica.cc:442] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:59.856451 8401 raft_consensus.cc:482] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:59.856544 8401 raft_consensus.cc:504] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.856681 8401 leader_election.cc:288] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697), 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747)
I20260620 09:16:59.856954 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6d932f2f084f89915100bc9cf066a4" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" is_pre_election: true
I20260620 09:16:59.856963 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6d932f2f084f89915100bc9cf066a4" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a65b02a291247b4ac5b5ad2477ace63" is_pre_election: true
I20260620 09:16:59.857052 8042 raft_consensus.cc:2454] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 0.
I20260620 09:16:59.857052 8322 raft_consensus.cc:2454] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 0.
I20260620 09:16:59.857218 8113 leader_election.cc:302] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a28a3d6afac4f1b9aa5fbf2494aaf89, 7f10df5b7fd74df2a3f237a89f1c176f; no voters:
I20260620 09:16:59.857342 8401 raft_consensus.cc:2781] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:59.857414 8401 raft_consensus.cc:482] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:59.857430 8401 raft_consensus.cc:3037] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:59.858132 8401 raft_consensus.cc:504] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.858268 8401 leader_election.cc:288] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 election: Requested vote from peers 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697), 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747)
I20260620 09:16:59.858417 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6d932f2f084f89915100bc9cf066a4" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f10df5b7fd74df2a3f237a89f1c176f"
I20260620 09:16:59.858424 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6d932f2f084f89915100bc9cf066a4" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a65b02a291247b4ac5b5ad2477ace63"
I20260620 09:16:59.858487 8042 raft_consensus.cc:3037] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:59.858492 8322 raft_consensus.cc:3037] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:59.859032 8042 raft_consensus.cc:2454] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 1.
I20260620 09:16:59.859099 8322 raft_consensus.cc:2454] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 1.
I20260620 09:16:59.859203 8113 leader_election.cc:302] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a28a3d6afac4f1b9aa5fbf2494aaf89, 7f10df5b7fd74df2a3f237a89f1c176f; no voters:
I20260620 09:16:59.859323 8401 raft_consensus.cc:2781] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:59.859385 8401 raft_consensus.cc:686] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 LEADER]: Becoming Leader. State: Replica: 6a28a3d6afac4f1b9aa5fbf2494aaf89, State: Running, Role: LEADER
I20260620 09:16:59.859458 8401 consensus_queue.cc:237] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.859814 8423 tablet_replica.cc:439] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: TxnStatusTablet state changed. Reason: New leader 6a28a3d6afac4f1b9aa5fbf2494aaf89. Latest consensus state: current_term: 1 leader_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } }
I20260620 09:16:59.859906 8423 tablet_replica.cc:442] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:16:59.860109 8435 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 09:16:59.860138 7885 catalog_manager.cc:5495] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a28a3d6afac4f1b9aa5fbf2494aaf89 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } health_report { overall_health: UNKNOWN } } }
I20260620 09:16:59.860229 8435 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 09:16:59.866359 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.863750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.0.98.2:39735)
0620 09:16:59.863787 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.863789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.863837 (+ 48us) server_negotiation.cc:407] Connection header received
0620 09:16:59.863928 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.863929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.863942 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.863995 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:59.864113 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.864578 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.865643 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.865816 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.865870 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.866069 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:59.866164 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:59.866230 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.866278 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.866297 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:16:59.866389 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.863714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39735 (local address 127.0.0.1:52594)
0620 09:16:59.863793 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.863804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.863857 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.864000 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.864001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.864002 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:16:59.864058 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.864060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.864640 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.864641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.865573 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.865576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.865885 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.865887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.865964 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.865977 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:16:59.866216 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:16:59.866223 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.866273 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.866323 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 09:16:59.867359 8042 raft_consensus.cc:1261] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Refusing update from remote peer 6a28a3d6afac4f1b9aa5fbf2494aaf89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:59.867411 8322 raft_consensus.cc:1261] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Refusing update from remote peer 6a28a3d6afac4f1b9aa5fbf2494aaf89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:59.867686 8401 consensus_queue.cc:1035] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:59.867808 8423 consensus_queue.cc:1035] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:59.868315 8399 tablet_replica.cc:439] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f: TxnStatusTablet state changed. Reason: New leader 6a28a3d6afac4f1b9aa5fbf2494aaf89. Latest consensus state: current_term: 1 leader_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } }
I20260620 09:16:59.868398 8399 tablet_replica.cc:442] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:59.868371 8403 tablet_replica.cc:439] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63: TxnStatusTablet state changed. Reason: New leader 6a28a3d6afac4f1b9aa5fbf2494aaf89. Latest consensus state: current_term: 1 leader_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } }
I20260620 09:16:59.868443 8403 tablet_replica.cc:442] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:59.869071 8401 tablet_replica.cc:439] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } }
I20260620 09:16:59.869138 8401 tablet_replica.cc:442] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:16:59.869323 8406 mvcc.cc:205] Tried to move back new op lower bound from 7298854993375105024 to 7298854993344819200. Current Snapshot: MvccSnapshot[applied={T|T < 7298854993375105024}]
I20260620 09:16:59.869403 8403 tablet_replica.cc:439] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } }
I20260620 09:16:59.869462 8403 tablet_replica.cc:442] T 7c6d932f2f084f89915100bc9cf066a4 P 9a65b02a291247b4ac5b5ad2477ace63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:59.870411 8423 tablet_replica.cc:439] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } }
I20260620 09:16:59.870491 8423 tablet_replica.cc:442] T 7c6d932f2f084f89915100bc9cf066a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:16:59.870738 8407 mvcc.cc:205] Tried to move back new op lower bound from 7298854993375105024 to 7298854993344819200. Current Snapshot: MvccSnapshot[applied={T|T < 7298854993375105024}]
I20260620 09:16:59.870820 8399 tablet_replica.cc:439] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } } }
I20260620 09:16:59.870903 8399 tablet_replica.cc:442] T 7c6d932f2f084f89915100bc9cf066a4 P 7f10df5b7fd74df2a3f237a89f1c176f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:16:59.873315 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.870586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.0.98.2:39735)
0620 09:16:59.870621 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.870623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.870630 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:59.870674 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.870675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.870684 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.870734 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:59.871231 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.871972 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.872668 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.872947 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.873018 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.873099 (+ 81us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:59.873129 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:59.873169 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.873235 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.873251 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 09:16:59.878084 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.875462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59231 (local address 127.0.98.2:39735)
0620 09:16:59.875495 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.875497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.875713 (+ 216us) server_negotiation.cc:407] Connection header received
0620 09:16:59.875737 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.875738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.875747 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.875801 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:59.875924 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.876369 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.877507 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.877679 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.877745 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.877899 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:59.877920 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:59.877951 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.878001 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.878020 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":16}
I20260620 09:16:59.878180 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.875420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39735 (local address 127.0.98.2:59231)
0620 09:16:59.875576 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:16:59.875585 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:16:59.875661 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:16:59.875805 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:16:59.875806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:16:59.875808 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:16:59.875869 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.875870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.876430 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.876431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.877420 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:16:59.877423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.877754 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.877755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:16:59.877820 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.877834 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:16:59.877954 (+ 120us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:16:59.877956 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:16:59.877999 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 09:16:59.878044 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260620 09:16:59.879962 8042 raft_consensus.cc:1261] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Refusing update from remote peer 6a28a3d6afac4f1b9aa5fbf2494aaf89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:59.879962 8322 raft_consensus.cc:1261] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Refusing update from remote peer 6a28a3d6afac4f1b9aa5fbf2494aaf89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:59.880139 8423 consensus_queue.cc:1035] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:59.880252 8440 consensus_queue.cc:1035] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:59.890612 7885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:59.890712 7885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:59.891243 7885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8: Sending DeleteTablet for 3 replicas of tablet 55bc08f2f3f14092bbb259214a8279a4
I20260620 09:16:59.891458 8302 tablet_service.cc:1514] Processing DeleteTablet for tablet 55bc08f2f3f14092bbb259214a8279a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53476
I20260620 09:16:59.891462 8022 tablet_service.cc:1514] Processing DeleteTablet for tablet 55bc08f2f3f14092bbb259214a8279a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44292
I20260620 09:16:59.891551 8415 tablet_replica.cc:330] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: stopping tablet replica
I20260620 09:16:59.891556 8417 tablet_replica.cc:330] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63: stopping tablet replica
I20260620 09:16:59.891597 8415 raft_consensus.cc:2229] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:59.891601 8417 raft_consensus.cc:2229] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:59.891654 8415 pending_rounds.cc:62] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Trying to abort 2 pending ops.
I20260620 09:16:59.891681 8417 raft_consensus.cc:2258] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:59.891678 8415 pending_rounds.cc:68] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298854993427173376 op_type: PARTICIPANT_OP participant_request { tablet_id: "55bc08f2f3f14092bbb259214a8279a4" op { type: BEGIN_TXN txn_id: 0 } }
I20260620 09:16:59.891752 8155 tablet_service.cc:1514] Processing DeleteTablet for tablet 55bc08f2f3f14092bbb259214a8279a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52588
I20260620 09:16:59.891789 8415 pending_rounds.cc:68] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298854992954916864 op_type: NO_OP noop_request { }
I20260620 09:16:59.891821 8415 raft_consensus.cc:2866] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 09:16:59.891844 8415 raft_consensus.cc:2258] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:59.891868 8416 tablet_replica.cc:330] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: stopping tablet replica
I20260620 09:16:59.891916 8416 raft_consensus.cc:2229] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:59.891991 8416 raft_consensus.cc:2258] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:59.892207 8417 ts_tablet_manager.cc:1905] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:59.892267 8416 ts_tablet_manager.cc:1905] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:59.893671 8417 ts_tablet_manager.cc:1918] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:16:59.893725 8417 log.cc:1196] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/55bc08f2f3f14092bbb259214a8279a4
W20260620 09:16:59.893793 8445 consensus_queue.cc:1159] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260620 09:16:59.893918 8417 ts_tablet_manager.cc:1939] T 55bc08f2f3f14092bbb259214a8279a4 P 9a65b02a291247b4ac5b5ad2477ace63: Deleting consensus metadata
I20260620 09:16:59.894027 8415 ts_tablet_manager.cc:1905] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:59.894115 8416 ts_tablet_manager.cc:1918] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:16:59.894167 8416 log.cc:1196] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/55bc08f2f3f14092bbb259214a8279a4
I20260620 09:16:59.894351 8416 ts_tablet_manager.cc:1939] T 55bc08f2f3f14092bbb259214a8279a4 P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Deleting consensus metadata
I20260620 09:16:59.894606 7869 catalog_manager.cc:4841] TS 6a28a3d6afac4f1b9aa5fbf2494aaf89 (127.0.98.2:39735): tablet 55bc08f2f3f14092bbb259214a8279a4 (table test-table [id=8fc5722dbfcf4bbdb6b16f89b74e59fc]) successfully deleted
I20260620 09:16:59.894698 7870 catalog_manager.cc:4841] TS 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747): tablet 55bc08f2f3f14092bbb259214a8279a4 (table test-table [id=8fc5722dbfcf4bbdb6b16f89b74e59fc]) successfully deleted
I20260620 09:16:59.895259 8415 ts_tablet_manager.cc:1918] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:59.895308 8415 log.cc:1196] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/wal/wals/55bc08f2f3f14092bbb259214a8279a4
I20260620 09:16:59.895473 8415 ts_tablet_manager.cc:1939] T 55bc08f2f3f14092bbb259214a8279a4 P 7f10df5b7fd74df2a3f237a89f1c176f: Deleting consensus metadata
I20260620 09:16:59.895846 7869 catalog_manager.cc:4841] TS 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697): tablet 55bc08f2f3f14092bbb259214a8279a4 (table test-table [id=8fc5722dbfcf4bbdb6b16f89b74e59fc]) successfully deleted
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:16:59.918491 7934 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:16:59.913071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.0.98.62:33893)
0620 09:16:59.913124 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.913126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.913228 (+ 102us) server_negotiation.cc:407] Connection header received
0620 09:16:59.913303 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.913304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.913314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.913361 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:59.913918 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.914419 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.915139 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.915349 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.915404 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.918417 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49526: BlockingRecv error: recv got EOF from 127.0.0.1:49526 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 09:16:59 dist-test-slave-rq8c krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:16:59.926236 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.919251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49536 (local address 127.0.98.62:33893)
0620 09:16:59.919307 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.919309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.919324 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:16:59.919386 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.919387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.919397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.919456 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:16:59.919913 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.920673 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.921379 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.921679 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.921744 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.923919 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:16:59.923921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:16:59.923921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:16:59.923928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:16:59.925292 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.925589 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.925590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.925590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.925641 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:16:59.925816 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:16:59.925817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:16:59.925817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:16:59.925930 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:16:59.926013 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.926082 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.926149 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:16:59.927771 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:59.927953 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:59.929198 8155 tablet_service.cc:1467] Processing CreateTablet for tablet 86577608d68d4874b9a8037d45bc3d6f (DEFAULT_TABLE table=test-table [id=c4f990feba524f79ba7b3071fa4c86d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:59.929208 8022 tablet_service.cc:1467] Processing CreateTablet for tablet 86577608d68d4874b9a8037d45bc3d6f (DEFAULT_TABLE table=test-table [id=c4f990feba524f79ba7b3071fa4c86d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:59.929210 8302 tablet_service.cc:1467] Processing CreateTablet for tablet 86577608d68d4874b9a8037d45bc3d6f (DEFAULT_TABLE table=test-table [id=c4f990feba524f79ba7b3071fa4c86d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:59.929325 8155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86577608d68d4874b9a8037d45bc3d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.929325 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86577608d68d4874b9a8037d45bc3d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.929328 8302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86577608d68d4874b9a8037d45bc3d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.930565 8393 tablet_bootstrap.cc:492] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f: Bootstrap starting.
I20260620 09:16:59.930576 8395 tablet_bootstrap.cc:492] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Bootstrap starting.
I20260620 09:16:59.930876 8395 tablet_bootstrap.cc:654] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.930919 8393 tablet_bootstrap.cc:654] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.931109 8394 tablet_bootstrap.cc:492] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63: Bootstrap starting.
I20260620 09:16:59.931408 8394 tablet_bootstrap.cc:654] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.931919 8394 tablet_bootstrap.cc:492] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63: No bootstrap required, opened a new log
I20260620 09:16:59.931972 8394 ts_tablet_manager.cc:1397] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:16:59.932078 8394 raft_consensus.cc:348] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.932147 8394 raft_consensus.cc:374] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.932164 8394 raft_consensus.cc:729] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a65b02a291247b4ac5b5ad2477ace63, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.932206 8394 consensus_queue.cc:260] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.932291 8394 ts_tablet_manager.cc:1428] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:59.932482 8395 tablet_bootstrap.cc:492] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89: No bootstrap required, opened a new log
I20260620 09:16:59.932524 8395 ts_tablet_manager.cc:1397] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:16:59.932663 8395 raft_consensus.cc:348] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.932731 8395 raft_consensus.cc:374] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.932749 8395 raft_consensus.cc:729] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a28a3d6afac4f1b9aa5fbf2494aaf89, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.932789 8395 consensus_queue.cc:260] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.932868 8395 ts_tablet_manager.cc:1428] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 09:16:59.933279 8393 tablet_bootstrap.cc:492] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f: No bootstrap required, opened a new log
I20260620 09:16:59.933321 8393 ts_tablet_manager.cc:1397] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 09:16:59.933462 8393 raft_consensus.cc:348] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.933523 8393 raft_consensus.cc:374] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.933542 8393 raft_consensus.cc:729] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f10df5b7fd74df2a3f237a89f1c176f, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.933581 8393 consensus_queue.cc:260] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.933661 8393 ts_tablet_manager.cc:1428] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:59.966216 8423 raft_consensus.cc:482] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:16:59.966323 8423 raft_consensus.cc:504] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.966492 8423 leader_election.cc:288] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697), 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747)
I20260620 09:16:59.966648 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86577608d68d4874b9a8037d45bc3d6f" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" is_pre_election: true
I20260620 09:16:59.966667 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86577608d68d4874b9a8037d45bc3d6f" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a65b02a291247b4ac5b5ad2477ace63" is_pre_election: true
I20260620 09:16:59.966750 8322 raft_consensus.cc:2454] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 0.
I20260620 09:16:59.966750 8042 raft_consensus.cc:2454] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 0.
I20260620 09:16:59.966939 8113 leader_election.cc:302] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a28a3d6afac4f1b9aa5fbf2494aaf89, 7f10df5b7fd74df2a3f237a89f1c176f; no voters:
I20260620 09:16:59.967026 8423 raft_consensus.cc:2781] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:16:59.967069 8423 raft_consensus.cc:482] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:16:59.967087 8423 raft_consensus.cc:3037] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:59.967940 8423 raft_consensus.cc:504] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.968065 8423 leader_election.cc:288] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 election: Requested vote from peers 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697), 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747)
I20260620 09:16:59.968176 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86577608d68d4874b9a8037d45bc3d6f" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f10df5b7fd74df2a3f237a89f1c176f"
I20260620 09:16:59.968250 8042 raft_consensus.cc:3037] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:59.968233 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86577608d68d4874b9a8037d45bc3d6f" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a65b02a291247b4ac5b5ad2477ace63"
I20260620 09:16:59.968297 8322 raft_consensus.cc:3037] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:16:59.968969 8042 raft_consensus.cc:2454] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 1.
I20260620 09:16:59.968981 8322 raft_consensus.cc:2454] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 1.
I20260620 09:16:59.969146 8113 leader_election.cc:302] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a28a3d6afac4f1b9aa5fbf2494aaf89, 7f10df5b7fd74df2a3f237a89f1c176f; no voters:
I20260620 09:16:59.969273 8423 raft_consensus.cc:2781] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:16:59.969336 8423 raft_consensus.cc:686] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 LEADER]: Becoming Leader. State: Replica: 6a28a3d6afac4f1b9aa5fbf2494aaf89, State: Running, Role: LEADER
I20260620 09:16:59.969376 8423 consensus_queue.cc:237] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.969900 7884 catalog_manager.cc:5495] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a28a3d6afac4f1b9aa5fbf2494aaf89 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } health_report { overall_health: UNKNOWN } } }
I20260620 09:16:59.982986 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:16:59.980611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.0.98.2:39735)
0620 09:16:59.980657 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:16:59.980659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:16:59.980666 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:16:59.980687 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:16:59.980688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:16:59.980696 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:16:59.980741 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:16:59.980879 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.981324 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.982389 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:16:59.982567 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:16:59.982619 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:16:59.982787 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:16:59.982812 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:16:59.982846 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 09:16:59.982912 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 09:16:59.982928 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:16:59.983670 8322 raft_consensus.cc:1261] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Refusing update from remote peer 6a28a3d6afac4f1b9aa5fbf2494aaf89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:59.983670 8042 raft_consensus.cc:1261] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Refusing update from remote peer 6a28a3d6afac4f1b9aa5fbf2494aaf89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:16:59.983956 8423 consensus_queue.cc:1035] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:59.984081 8401 consensus_queue.cc:1035] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:16:59.986398 7885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:59.986481 7885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:16:59.987067 7885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8: Sending DeleteTablet for 3 replicas of tablet 86577608d68d4874b9a8037d45bc3d6f
I20260620 09:16:59.987264 8155 tablet_service.cc:1514] Processing DeleteTablet for tablet 86577608d68d4874b9a8037d45bc3d6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52588
I20260620 09:16:59.987355 8416 tablet_replica.cc:330] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89: stopping tablet replica
I20260620 09:16:59.987325 8021 tablet_service.cc:1514] Processing DeleteTablet for tablet 86577608d68d4874b9a8037d45bc3d6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44292
I20260620 09:16:59.987330 8302 tablet_service.cc:1514] Processing DeleteTablet for tablet 86577608d68d4874b9a8037d45bc3d6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53476
I20260620 09:16:59.987396 8416 raft_consensus.cc:2229] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:16:59.987414 8417 tablet_replica.cc:330] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63: stopping tablet replica
I20260620 09:16:59.987389 8022 tablet_service.cc:1514] Processing DeleteTablet for tablet 86577608d68d4874b9a8037d45bc3d6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:16:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44292
I20260620 09:16:59.987448 8416 raft_consensus.cc:2258] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:59.987468 8415 tablet_replica.cc:330] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f: stopping tablet replica
I20260620 09:16:59.987458 8417 raft_consensus.cc:2229] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:59.987514 8417 raft_consensus.cc:2258] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:59.987509 8415 raft_consensus.cc:2229] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:16:59.987546 8415 raft_consensus.cc:2258] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:16:59.987946 8417 ts_tablet_manager.cc:1905] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:59.987949 8415 ts_tablet_manager.cc:1905] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:59.987962 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:16:59.988070 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:16:59.989377 8415 ts_tablet_manager.cc:1918] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:59.989391 8417 ts_tablet_manager.cc:1918] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:59.989428 8415 log.cc:1196] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-0/wal/wals/86577608d68d4874b9a8037d45bc3d6f
I20260620 09:16:59.989441 8417 log.cc:1196] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/86577608d68d4874b9a8037d45bc3d6f
I20260620 09:16:59.989630 8415 ts_tablet_manager.cc:1939] T 86577608d68d4874b9a8037d45bc3d6f P 7f10df5b7fd74df2a3f237a89f1c176f: Deleting consensus metadata
I20260620 09:16:59.989636 8417 ts_tablet_manager.cc:1939] T 86577608d68d4874b9a8037d45bc3d6f P 9a65b02a291247b4ac5b5ad2477ace63: Deleting consensus metadata
I20260620 09:16:59.989995 7870 catalog_manager.cc:4841] TS 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747): tablet 86577608d68d4874b9a8037d45bc3d6f (table test-table [id=c4f990feba524f79ba7b3071fa4c86d6]) successfully deleted
I20260620 09:16:59.990095 7869 catalog_manager.cc:4841] TS 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697): tablet 86577608d68d4874b9a8037d45bc3d6f (table test-table [id=c4f990feba524f79ba7b3071fa4c86d6]) successfully deleted
W20260620 09:16:59.990211 7869 catalog_manager.cc:4808] TS 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697): delete failed for tablet 86577608d68d4874b9a8037d45bc3d6f because the tablet was not found. No further retry: Not found: Tablet not found: 86577608d68d4874b9a8037d45bc3d6f
I20260620 09:16:59.990584 8416 ts_tablet_manager.cc:1905] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:16:59.990801 8302 tablet_service.cc:1467] Processing CreateTablet for tablet 6f782ec027ba4c359ce07439eeb06b4a (DEFAULT_TABLE table=test-table [id=b675da6f10984b4e869903e0d32bf5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:59.990844 8022 tablet_service.cc:1467] Processing CreateTablet for tablet 6f782ec027ba4c359ce07439eeb06b4a (DEFAULT_TABLE table=test-table [id=b675da6f10984b4e869903e0d32bf5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:59.990900 8302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f782ec027ba4c359ce07439eeb06b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.990942 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f782ec027ba4c359ce07439eeb06b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.991894 8416 ts_tablet_manager.cc:1918] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:16:59.991940 8416 log.cc:1196] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/86577608d68d4874b9a8037d45bc3d6f
I20260620 09:16:59.992043 8393 tablet_bootstrap.cc:492] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f: Bootstrap starting.
I20260620 09:16:59.992065 8394 tablet_bootstrap.cc:492] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63: Bootstrap starting.
I20260620 09:16:59.992112 8416 ts_tablet_manager.cc:1939] T 86577608d68d4874b9a8037d45bc3d6f P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Deleting consensus metadata
I20260620 09:16:59.992331 8394 tablet_bootstrap.cc:654] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.992398 8393 tablet_bootstrap.cc:654] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.992777 8394 tablet_bootstrap.cc:492] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63: No bootstrap required, opened a new log
I20260620 09:16:59.992817 8394 ts_tablet_manager.cc:1397] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:16:59.992815 7869 catalog_manager.cc:4841] TS 6a28a3d6afac4f1b9aa5fbf2494aaf89 (127.0.98.2:39735): tablet 86577608d68d4874b9a8037d45bc3d6f (table test-table [id=c4f990feba524f79ba7b3071fa4c86d6]) successfully deleted
I20260620 09:16:59.992900 8155 tablet_service.cc:1467] Processing CreateTablet for tablet 6f782ec027ba4c359ce07439eeb06b4a (DEFAULT_TABLE table=test-table [id=b675da6f10984b4e869903e0d32bf5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:16:59.992902 8394 raft_consensus.cc:348] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.992957 8394 raft_consensus.cc:374] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.992973 8155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f782ec027ba4c359ce07439eeb06b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:16:59.992973 8394 raft_consensus.cc:729] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a65b02a291247b4ac5b5ad2477ace63, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.993022 8394 consensus_queue.cc:260] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.993098 8394 ts_tablet_manager.cc:1428] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:16:59.994038 8395 tablet_bootstrap.cc:492] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Bootstrap starting.
I20260620 09:16:59.994370 8393 tablet_bootstrap.cc:492] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f: No bootstrap required, opened a new log
I20260620 09:16:59.994424 8395 tablet_bootstrap.cc:654] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Neither blocks nor log segments found. Creating new log.
I20260620 09:16:59.994434 8393 ts_tablet_manager.cc:1397] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:16:59.994586 8393 raft_consensus.cc:348] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.994656 8393 raft_consensus.cc:374] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.994678 8393 raft_consensus.cc:729] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f10df5b7fd74df2a3f237a89f1c176f, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.994721 8393 consensus_queue.cc:260] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.994814 8393 ts_tablet_manager.cc:1428] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 09:16:59.994927 8395 tablet_bootstrap.cc:492] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89: No bootstrap required, opened a new log
I20260620 09:16:59.994971 8395 ts_tablet_manager.cc:1397] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:16:59.995112 8395 raft_consensus.cc:348] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.995186 8395 raft_consensus.cc:374] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:16:59.995213 8395 raft_consensus.cc:729] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a28a3d6afac4f1b9aa5fbf2494aaf89, State: Initialized, Role: FOLLOWER
I20260620 09:16:59.995249 8395 consensus_queue.cc:260] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:16:59.995347 8395 ts_tablet_manager.cc:1428] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:00.372351 8423 raft_consensus.cc:482] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:17:00.372444 8423 raft_consensus.cc:504] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:17:00.372572 8423 leader_election.cc:288] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697), 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747)
I20260620 09:17:00.372797 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f782ec027ba4c359ce07439eeb06b4a" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a65b02a291247b4ac5b5ad2477ace63" is_pre_election: true
I20260620 09:17:00.372766 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f782ec027ba4c359ce07439eeb06b4a" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" is_pre_election: true
I20260620 09:17:00.372879 8322 raft_consensus.cc:2454] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 0.
I20260620 09:17:00.372885 8042 raft_consensus.cc:2454] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 0.
I20260620 09:17:00.373028 8114 leader_election.cc:302] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a28a3d6afac4f1b9aa5fbf2494aaf89, 9a65b02a291247b4ac5b5ad2477ace63; no voters:
I20260620 09:17:00.373169 8423 raft_consensus.cc:2781] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:17:00.373217 8423 raft_consensus.cc:482] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:17:00.373241 8423 raft_consensus.cc:3037] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:17:00.373909 8423 raft_consensus.cc:504] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:17:00.374038 8423 leader_election.cc:288] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 election: Requested vote from peers 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697), 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747)
I20260620 09:17:00.374166 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f782ec027ba4c359ce07439eeb06b4a" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f10df5b7fd74df2a3f237a89f1c176f"
I20260620 09:17:00.374248 8042 raft_consensus.cc:3037] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:17:00.374210 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f782ec027ba4c359ce07439eeb06b4a" candidate_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a65b02a291247b4ac5b5ad2477ace63"
I20260620 09:17:00.374306 8322 raft_consensus.cc:3037] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:17:00.374837 8042 raft_consensus.cc:2454] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 1.
I20260620 09:17:00.374965 8322 raft_consensus.cc:2454] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a28a3d6afac4f1b9aa5fbf2494aaf89 in term 1.
I20260620 09:17:00.375010 8113 leader_election.cc:302] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a28a3d6afac4f1b9aa5fbf2494aaf89, 7f10df5b7fd74df2a3f237a89f1c176f; no voters:
I20260620 09:17:00.375075 8423 raft_consensus.cc:2781] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:17:00.375123 8423 raft_consensus.cc:686] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 LEADER]: Becoming Leader. State: Replica: 6a28a3d6afac4f1b9aa5fbf2494aaf89, State: Running, Role: LEADER
I20260620 09:17:00.375167 8423 consensus_queue.cc:237] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } }
I20260620 09:17:00.375793 7884 catalog_manager.cc:5495] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a28a3d6afac4f1b9aa5fbf2494aaf89 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a28a3d6afac4f1b9aa5fbf2494aaf89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 } health_report { overall_health: UNKNOWN } } }
I20260620 09:17:00.437974 8042 raft_consensus.cc:1261] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Refusing update from remote peer 6a28a3d6afac4f1b9aa5fbf2494aaf89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:17:00.438012 8322 raft_consensus.cc:1261] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Refusing update from remote peer 6a28a3d6afac4f1b9aa5fbf2494aaf89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:17:00.438194 8423 consensus_queue.cc:1035] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f10df5b7fd74df2a3f237a89f1c176f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:17:00.438313 8401 consensus_queue.cc:1035] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a65b02a291247b4ac5b5ad2477ace63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:17:00.447301 7885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:17:00.447391 7885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:17:00.447901 7885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a6740cf374346938d9d6abbf1e913d8: Sending DeleteTablet for 3 replicas of tablet 6f782ec027ba4c359ce07439eeb06b4a
I20260620 09:17:00.448120 8022 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f782ec027ba4c359ce07439eeb06b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:17:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44292
I20260620 09:17:00.448114 8302 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f782ec027ba4c359ce07439eeb06b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:17:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53476
I20260620 09:17:00.448204 8415 tablet_replica.cc:330] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f: stopping tablet replica
I20260620 09:17:00.448223 8417 tablet_replica.cc:330] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63: stopping tablet replica
I20260620 09:17:00.448248 8415 raft_consensus.cc:2229] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:17:00.448282 8415 raft_consensus.cc:2258] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:17:00.448299 8417 raft_consensus.cc:2229] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:17:00.448343 8417 raft_consensus.cc:2258] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:17:00.448575 8415 ts_tablet_manager.cc:1905] T 6f782ec027ba4c359ce07439eeb06b4a P 7f10df5b7fd74df2a3f237a89f1c176f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:17:00.448652 8417 ts_tablet_manager.cc:1905] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:17:00.448679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 7957
I20260620 09:17:00.450004 8417 ts_tablet_manager.cc:1918] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:17:00.450058 8417 log.cc:1196] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/6f782ec027ba4c359ce07439eeb06b4a
I20260620 09:17:00.450233 8417 ts_tablet_manager.cc:1939] T 6f782ec027ba4c359ce07439eeb06b4a P 9a65b02a291247b4ac5b5ad2477ace63: Deleting consensus metadata
I20260620 09:17:00.450512 7870 catalog_manager.cc:4841] TS 9a65b02a291247b4ac5b5ad2477ace63 (127.0.98.3:46747): tablet 6f782ec027ba4c359ce07439eeb06b4a (table test-table [id=b675da6f10984b4e869903e0d32bf5f5]) successfully deleted
W20260620 09:17:00.453758 7869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38697 (error 108)
I20260620 09:17:00.453827 8155 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f782ec027ba4c359ce07439eeb06b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:17:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52588
I20260620 09:17:00.453879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 8097
I20260620 09:17:00.453926 8416 tablet_replica.cc:330] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89: stopping tablet replica
I20260620 09:17:00.453970 8416 raft_consensus.cc:2229] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:17:00.454035 8416 raft_consensus.cc:2258] T 6f782ec027ba4c359ce07439eeb06b4a P 6a28a3d6afac4f1b9aa5fbf2494aaf89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:17:00.455014 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:00.454845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38697 (local address 127.0.0.1:44298)
0620 09:17:00.454939 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:00.454982 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":40,"threads_started":1}
W20260620 09:17:00.455152 7869 catalog_manager.cc:4584] TS 7f10df5b7fd74df2a3f237a89f1c176f (127.0.98.1:38697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f782ec027ba4c359ce07439eeb06b4a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38697: connect: Connection refused (error 111)
I20260620 09:17:00.459059 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:00.458958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39735 (local address 127.0.0.1:52612)
0620 09:17:00.458999 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:00.459023 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 09:17:00.459061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 8237
I20260620 09:17:00.463445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 7853
2026-06-20T09:17:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2865 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:17:00 dist-test-slave-rq8c krb5kdc[8478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:17:00 dist-test-slave-rq8c krb5kdc[8478](info): set up 2 sockets
Jun 20 09:17:00 dist-test-slave-rq8c krb5kdc[8478](info): commencing operation
krb5kdc: starting...
W20260620 09:17:01.415957 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:17:01 dist-test-slave-rq8c krb5kdc[8478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:17:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:17:01.523870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44415
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:17:01.601594 8491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:01.601804 8491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:01.601833 8491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:01.602943 8491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:17:01.602981 8491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:01.602999 8491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:01.603015 8491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:17:01.603031 8491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:17:01.604547 8491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:01.604779 8491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:01.604976 8491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:01.609453 8499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:01.609479 8491 server_base.cc:1034] running on GCE node
W20260620 09:17:01.609480 8496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:01.609454 8497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:01.609824 8491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:01.610065 8491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:01.611212 8491 hybrid_clock.cc:648] HybridClock initialized: now 1781947021611198 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T09:17:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1158 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:17:01 dist-test-slave-rq8c krb5kdc[8504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:17:01 dist-test-slave-rq8c krb5kdc[8504](info): set up 2 sockets
Jun 20 09:17:01 dist-test-slave-rq8c krb5kdc[8504](info): commencing operation
krb5kdc: starting...
W20260620 09:17:02.583879 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:17:02 dist-test-slave-rq8c krb5kdc[8504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:17:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:17:02.691250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44519
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:17:02.769184 8517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:02.769405 8517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:02.769436 8517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:02.770586 8517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:17:02.770627 8517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:02.770645 8517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:02.770663 8517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:17:02.770678 8517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:17:02.772207 8517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:02.772465 8517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:02.772672 8517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:02.777356 8522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:02.777410 8523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:02.777446 8517 server_base.cc:1034] running on GCE node
W20260620 09:17:02.777424 8525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:02.777796 8517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:02.778057 8517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:02.779210 8517 hybrid_clock.cc:648] HybridClock initialized: now 1781947022779191 us; error 36 us; skew 500 ppm
Jun 20 09:17:02 dist-test-slave-rq8c krb5kdc[8504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:17:02.781531 8517 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:17:02.781925 8517 webserver.cc:460] Webserver started at http://127.0.98.62:36935/ using document root <none> and password file <none>
I20260620 09:17:02.782131 8517 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:02.782178 8517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:02.782297 8517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:02.783327 8517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "fb9f1d7b081248fa98a31a13c3432a4c"
format_stamp: "Formatted at 2026-06-20 09:17:02 on dist-test-slave-rq8c"
server_key: "9292a68726688d31088a1e0eb50c92a3"
server_key_iv: "e70dec9e678ffe3ad1a6aee648c7e1a0"
server_key_version: "encryptionkey@0"
I20260620 09:17:02.783708 8517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb9f1d7b081248fa98a31a13c3432a4c"
format_stamp: "Formatted at 2026-06-20 09:17:02 on dist-test-slave-rq8c"
server_key: "9292a68726688d31088a1e0eb50c92a3"
server_key_iv: "e70dec9e678ffe3ad1a6aee648c7e1a0"
server_key_version: "encryptionkey@0"
I20260620 09:17:02.784909 8517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:17:02.785591 8532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:02.785737 8517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:17:02.785817 8517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "fb9f1d7b081248fa98a31a13c3432a4c"
format_stamp: "Formatted at 2026-06-20 09:17:02 on dist-test-slave-rq8c"
server_key: "9292a68726688d31088a1e0eb50c92a3"
server_key_iv: "e70dec9e678ffe3ad1a6aee648c7e1a0"
server_key_version: "encryptionkey@0"
I20260620 09:17:02.785880 8517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:02.809700 8517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:02.811532 8517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:02.811709 8517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:02.815272 8517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44519
I20260620 09:17:02.815328 8584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44519 every 8 connection(s)
I20260620 09:17:02.815799 8517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:17:02.816449 8585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:17:02.818638 8585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c: Bootstrap starting.
I20260620 09:17:02.819183 8585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c: Neither blocks nor log segments found. Creating new log.
I20260620 09:17:02.819448 8585 log.cc:824] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:17:02.820093 8585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c: No bootstrap required, opened a new log
I20260620 09:17:02.821612 8585 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44519 } }
I20260620 09:17:02.821748 8585 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:17:02.821779 8585 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9f1d7b081248fa98a31a13c3432a4c, State: Initialized, Role: FOLLOWER
I20260620 09:17:02.821919 8585 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44519 } }
I20260620 09:17:02.821987 8585 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:17:02.822021 8585 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:17:02.822064 8585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:17:02.822885 8585 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44519 } }
I20260620 09:17:02.822975 8585 leader_election.cc:302] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb9f1d7b081248fa98a31a13c3432a4c; no voters:
I20260620 09:17:02.823125 8585 leader_election.cc:288] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:17:02.823166 8588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:17:02.823307 8588 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 1 LEADER]: Becoming Leader. State: Replica: fb9f1d7b081248fa98a31a13c3432a4c, State: Running, Role: LEADER
I20260620 09:17:02.823354 8585 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:17:02.823391 8588 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44519 } }
I20260620 09:17:02.823745 8590 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb9f1d7b081248fa98a31a13c3432a4c. Latest consensus state: current_term: 1 leader_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44519 } } }
I20260620 09:17:02.823838 8590 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [sys.catalog]: This master's current role is: LEADER
I20260620 09:17:02.823741 8589 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44519 } } }
I20260620 09:17:02.823958 8589 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [sys.catalog]: This master's current role is: LEADER
I20260620 09:17:02.824326 8595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:17:02.824820 8595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:17:02.825596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 8517
I20260620 09:17:02.825711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:17:02.825899 392 external_mini_cluster.cc:1428] Setting key b8b88cad0c42a71b22a034249f26b889
I20260620 09:17:02.826308 8595 catalog_manager.cc:1347] Generated new cluster ID: 4eb59c23a5d34f14b5fc390f0c778f53
I20260620 09:17:02.826354 8595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:17:02 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:17:02.834805 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:02.826492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.0.98.62:44519)
0620 09:17:02.826651 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:02.826682 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:02.826717 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:17:02.826806 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:02.826811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:02.826861 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:02.826973 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:02.827562 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:02.828550 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:02.829268 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:02.829697 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:02.829796 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:02.831906 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:02.831910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:02.831910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:17:02.831922 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:02.833401 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:02.833705 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:02.833708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:02.833709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:02.833789 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:02.833953 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:02.833953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:02.833954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:02.834191 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:02.834329 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:02.834432 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 09:17:02.834648 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260620 09:17:02.838682 8595 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:17:02.839144 8595 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:17:02.844404 8595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c: Generated new TSK 0
I20260620 09:17:02.844554 8595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:17:02.871605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44519
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:17:02.950387 8611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:02.950583 8611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:02.950605 8611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:02.951757 8611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:02.951794 8611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:02.951819 8611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:17:02.953279 8611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:02.953481 8611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:02.953682 8611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:02.958225 8616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:02.958256 8617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:02.958272 8619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:02.958366 8611 server_base.cc:1034] running on GCE node
I20260620 09:17:02.958693 8611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:02.958907 8611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:02.960034 8611 hybrid_clock.cc:648] HybridClock initialized: now 1781947022960023 us; error 33 us; skew 500 ppm
Jun 20 09:17:02 dist-test-slave-rq8c krb5kdc[8504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:17:02.962337 8611 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:17:02.962724 8611 webserver.cc:460] Webserver started at http://127.0.98.1:35883/ using document root <none> and password file <none>
I20260620 09:17:02.962905 8611 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:02.962950 8611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:02.963068 8611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:02.964048 8611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "946a87e49e7d4656b6e2672fcd036afd"
format_stamp: "Formatted at 2026-06-20 09:17:02 on dist-test-slave-rq8c"
server_key: "8a9b47b41b33d1143c23b1fbdb6efd84"
server_key_iv: "73ada0759136752b7d90e01a4a547e36"
server_key_version: "encryptionkey@0"
I20260620 09:17:02.964366 8611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "946a87e49e7d4656b6e2672fcd036afd"
format_stamp: "Formatted at 2026-06-20 09:17:02 on dist-test-slave-rq8c"
server_key: "8a9b47b41b33d1143c23b1fbdb6efd84"
server_key_iv: "73ada0759136752b7d90e01a4a547e36"
server_key_version: "encryptionkey@0"
I20260620 09:17:02.965492 8611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:17:02.966102 8626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:02.966246 8611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:17:02.966323 8611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "946a87e49e7d4656b6e2672fcd036afd"
format_stamp: "Formatted at 2026-06-20 09:17:02 on dist-test-slave-rq8c"
server_key: "8a9b47b41b33d1143c23b1fbdb6efd84"
server_key_iv: "73ada0759136752b7d90e01a4a547e36"
server_key_version: "encryptionkey@0"
I20260620 09:17:02.966378 8611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:02.984403 8611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:02.986435 8611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:02.986563 8611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:02.986769 8611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:02.987051 8611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:02.987085 8611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:02.987113 8611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:02.987138 8611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:02.992383 8611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40957
I20260620 09:17:02.992434 8739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40957 every 8 connection(s)
I20260620 09:17:02.992713 8611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:17:02.996117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 8611
I20260620 09:17:02.996209 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:17:02.996366 392 external_mini_cluster.cc:1428] Setting key a0b16d9e3119fb3e16099bd1f144d7ae
Jun 20 09:17:02 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:17:03.002399 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:02.993420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54205 (local address 127.0.98.62:44519)
0620 09:17:02.993472 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:02.993474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:02.993660 (+ 186us) server_negotiation.cc:407] Connection header received
0620 09:17:02.994554 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:02.994555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:02.994566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:02.994622 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:02.995185 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:02.995752 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:02.996807 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:02.997064 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:02.997124 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:02.999312 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:02.999313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:02.999314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:17:02.999321 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:03.000775 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:03.001263 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:03.001264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:03.001264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:03.001326 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:03.001695 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:03.001696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:03.001696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:03.001822 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:03.001889 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:03.002252 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 09:17:03.002332 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:17:03.002619 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:02.993383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:54205)
0620 09:17:02.993548 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:02.993587 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:02.994453 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:02.994693 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:02.994696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:02.994924 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:02.995103 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:02.995106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:02.995821 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:02.995822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:02.996739 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:02.996741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:02.997136 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:02.997137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:02.997261 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:02.997697 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:02.997705 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:02.999197 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:17:03.000865 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:17:03.000873 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:17:03.000876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:17:03.001171 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:17:03.001400 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:17:03.001400 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:17:03.001401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:17:03.001540 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:17:03.001895 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:17:03.001901 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:17:03.002076 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 09:17:03.002205 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 09:17:03.002376 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260620 09:17:03.003113 8740 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44519
I20260620 09:17:03.003214 8740 heartbeater.cc:464] Registering TS with master...
I20260620 09:17:03.003409 8740 heartbeater.cc:511] Master 127.0.98.62:44519 requested a full tablet report, sending...
I20260620 09:17:03.003942 8549 ts_manager.cc:194] Registered new tserver with Master: 946a87e49e7d4656b6e2672fcd036afd (127.0.98.1:40957)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:17:03.004539 8549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:17:03.018617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44519
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:17:03.097867 8745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:03.098120 8745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:03.098155 8745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:03.099463 8745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:03.099505 8745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:03.099539 8745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:17:03.100999 8745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:03.101227 8745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:03.101428 8745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:03.106099 8750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:03.106212 8745 server_base.cc:1034] running on GCE node
W20260620 09:17:03.106125 8753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:03.106112 8751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:03.106539 8745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:03.106756 8745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:03.107887 8745 hybrid_clock.cc:648] HybridClock initialized: now 1781947023107873 us; error 32 us; skew 500 ppm
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:17:03.110186 8745 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:17:03.110581 8745 webserver.cc:460] Webserver started at http://127.0.98.2:36777/ using document root <none> and password file <none>
I20260620 09:17:03.110764 8745 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:03.110808 8745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:03.110929 8745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:03.111922 8745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "8a5a91470e494c349cf15a27b5efac9a"
format_stamp: "Formatted at 2026-06-20 09:17:03 on dist-test-slave-rq8c"
server_key: "d613da93cd53e530b31d82a93f496196"
server_key_iv: "3f8b2eb637e20dde6de8493be5442c0e"
server_key_version: "encryptionkey@0"
I20260620 09:17:03.112258 8745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8a5a91470e494c349cf15a27b5efac9a"
format_stamp: "Formatted at 2026-06-20 09:17:03 on dist-test-slave-rq8c"
server_key: "d613da93cd53e530b31d82a93f496196"
server_key_iv: "3f8b2eb637e20dde6de8493be5442c0e"
server_key_version: "encryptionkey@0"
I20260620 09:17:03.113416 8745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:17:03.114018 8760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.114182 8745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:17:03.114259 8745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "8a5a91470e494c349cf15a27b5efac9a"
format_stamp: "Formatted at 2026-06-20 09:17:03 on dist-test-slave-rq8c"
server_key: "d613da93cd53e530b31d82a93f496196"
server_key_iv: "3f8b2eb637e20dde6de8493be5442c0e"
server_key_version: "encryptionkey@0"
I20260620 09:17:03.114316 8745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:03.122679 8745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:03.124778 8745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:03.124898 8745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:03.125097 8745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:03.125387 8745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:03.125420 8745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.125448 8745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:03.125473 8745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.130724 8745 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46515
I20260620 09:17:03.130765 8873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46515 every 8 connection(s)
I20260620 09:17:03.131037 8745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:17:03.132767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 8745
I20260620 09:17:03.132871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:17:03.133059 392 external_mini_cluster.cc:1428] Setting key fc39f0b9e779cf1a9937a88315634bbc
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:17:03.140434 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.131717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60215 (local address 127.0.98.62:44519)
0620 09:17:03.131770 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:03.131773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:03.131947 (+ 174us) server_negotiation.cc:407] Connection header received
0620 09:17:03.132833 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:03.132835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:03.132848 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:03.132913 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:03.133727 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.134400 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.135355 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.135543 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.135607 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.137875 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:03.137876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:03.137877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:17:03.137883 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:03.139084 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:03.139509 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:03.139509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:03.139510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:03.139554 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:03.139829 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:03.139830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:03.139830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:03.139935 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:03.139988 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:03.140307 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 09:17:03.140371 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:17:03.140659 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.131680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:60215)
0620 09:17:03.131836 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:03.131869 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:03.132739 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:03.133007 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:03.133010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:03.133321 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:03.133553 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.133560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.134474 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.134475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.135266 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.135268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.135612 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.135632 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.135777 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.136216 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:03.136225 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:03.137786 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:17:03.139161 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:17:03.139168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:17:03.139170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:17:03.139441 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:17:03.139608 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:17:03.139609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:17:03.139609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:17:03.139758 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:17:03.139998 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:17:03.140003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:17:03.140173 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 09:17:03.140285 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 09:17:03.140441 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:17:03.141160 8874 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44519
I20260620 09:17:03.141237 8874 heartbeater.cc:464] Registering TS with master...
I20260620 09:17:03.141371 8874 heartbeater.cc:511] Master 127.0.98.62:44519 requested a full tablet report, sending...
I20260620 09:17:03.141685 8549 ts_manager.cc:194] Registered new tserver with Master: 8a5a91470e494c349cf15a27b5efac9a (127.0.98.2:46515)
I20260620 09:17:03.142033 8549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:17:03.155141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44519
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:17:03.233527 8879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:03.233732 8879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:03.233754 8879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:03.234843 8879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:03.234876 8879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:03.234900 8879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:17:03.236357 8879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:03.236534 8879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:03.236723 8879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:03.241365 8884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:03.241382 8887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:03.241420 8885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:03.241407 8879 server_base.cc:1034] running on GCE node
I20260620 09:17:03.241683 8879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:03.241899 8879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:03.243031 8879 hybrid_clock.cc:648] HybridClock initialized: now 1781947023242999 us; error 52 us; skew 500 ppm
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:17:03.245375 8879 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:17:03.245792 8879 webserver.cc:460] Webserver started at http://127.0.98.3:43669/ using document root <none> and password file <none>
I20260620 09:17:03.245977 8879 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:03.246021 8879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:03.246145 8879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:03.247110 8879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "740360835dc0467d9c3ae6c351007cd9"
format_stamp: "Formatted at 2026-06-20 09:17:03 on dist-test-slave-rq8c"
server_key: "c19f5e5a07f51b6a7bbb9b8443fbf5a5"
server_key_iv: "246b7e4c0450979b3fa8bb4d5b0d9f21"
server_key_version: "encryptionkey@0"
I20260620 09:17:03.247428 8879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "740360835dc0467d9c3ae6c351007cd9"
format_stamp: "Formatted at 2026-06-20 09:17:03 on dist-test-slave-rq8c"
server_key: "c19f5e5a07f51b6a7bbb9b8443fbf5a5"
server_key_iv: "246b7e4c0450979b3fa8bb4d5b0d9f21"
server_key_version: "encryptionkey@0"
I20260620 09:17:03.248646 8879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:17:03.249264 8894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.249403 8879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.249480 8879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "740360835dc0467d9c3ae6c351007cd9"
format_stamp: "Formatted at 2026-06-20 09:17:03 on dist-test-slave-rq8c"
server_key: "c19f5e5a07f51b6a7bbb9b8443fbf5a5"
server_key_iv: "246b7e4c0450979b3fa8bb4d5b0d9f21"
server_key_version: "encryptionkey@0"
I20260620 09:17:03.249542 8879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:03.267282 8879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:03.269122 8879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:03.269251 8879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:03.269456 8879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:03.269742 8879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:03.269773 8879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.269802 8879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:03.269825 8879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.275063 8879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33371
I20260620 09:17:03.275115 9007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33371 every 8 connection(s)
I20260620 09:17:03.275393 8879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:17:03.279366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 8879
I20260620 09:17:03.279470 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:17:03.279675 392 external_mini_cluster.cc:1428] Setting key ebb574702ddf31405191b1ae69d1df8f
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:17:03.283421 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.276070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35743 (local address 127.0.98.62:44519)
0620 09:17:03.276141 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:03.276143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:03.276284 (+ 141us) server_negotiation.cc:407] Connection header received
0620 09:17:03.277181 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:03.277182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:03.277192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:03.277232 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:03.277784 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.278270 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.279180 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.279348 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.279391 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.281405 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:03.281407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:03.281407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:17:03.281412 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:03.282440 (+ 1028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:03.282757 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:03.282757 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:03.282758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:03.282798 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:03.282963 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:03.282963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:03.282964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:03.283053 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:03.283106 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:03.283306 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 09:17:03.283359 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:17:03.283591 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.276031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:35743)
0620 09:17:03.276178 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:03.276212 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:03.277090 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:03.277289 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:03.277292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:03.277532 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:03.277724 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.277728 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.278327 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.278328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.279124 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.279126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.279415 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.279415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.279571 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.280060 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:03.280068 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:03.281319 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:17:03.282498 (+ 1179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:17:03.282503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:17:03.282504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:17:03.282704 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:17:03.282840 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:17:03.282841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:17:03.282841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:17:03.282924 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:17:03.283103 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:17:03.283106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:17:03.283212 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 09:17:03.283287 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:17:03.283402 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260620 09:17:03.283995 9008 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44519
I20260620 09:17:03.284085 9008 heartbeater.cc:464] Registering TS with master...
I20260620 09:17:03.284245 9008 heartbeater.cc:511] Master 127.0.98.62:44519 requested a full tablet report, sending...
I20260620 09:17:03.284557 8549 ts_manager.cc:194] Registered new tserver with Master: 740360835dc0467d9c3ae6c351007cd9 (127.0.98.3:33371)
I20260620 09:17:03.284889 8549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35743
I20260620 09:17:03.291285 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 09:17:03.352063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 8611
I20260620 09:17:03.356351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 8745
I20260620 09:17:03.359949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 8879
I20260620 09:17:03.363427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 8517
I20260620 09:17:03.367811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44519
--webserver_interface=127.0.98.62
--webserver_port=36935
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:17:03.446187 9017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:03.446376 9017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:03.446396 9017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:03.447528 9017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:17:03.447567 9017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:03.447579 9017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:03.447599 9017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:17:03.447649 9017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:17:03.449129 9017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:03.449344 9017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:03.449545 9017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:03.453920 9022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:03.453969 9023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:03.453926 9025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:03.454214 9017 server_base.cc:1034] running on GCE node
I20260620 09:17:03.454406 9017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:03.454635 9017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:03.455757 9017 hybrid_clock.cc:648] HybridClock initialized: now 1781947023455742 us; error 31 us; skew 500 ppm
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:17:03.458077 9017 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:17:03.458456 9017 webserver.cc:460] Webserver started at http://127.0.98.62:36935/ using document root <none> and password file <none>
I20260620 09:17:03.458666 9017 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:03.458726 9017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:03.460052 9017 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:17:03.460652 9032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.460803 9017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:17:03.460881 9017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "fb9f1d7b081248fa98a31a13c3432a4c"
format_stamp: "Formatted at 2026-06-20 09:17:02 on dist-test-slave-rq8c"
server_key: "9292a68726688d31088a1e0eb50c92a3"
server_key_iv: "e70dec9e678ffe3ad1a6aee648c7e1a0"
server_key_version: "encryptionkey@0"
I20260620 09:17:03.461185 9017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:03.476047 9017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:03.477970 9017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:03.478104 9017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:03.481696 9017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44519
I20260620 09:17:03.481741 9084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44519 every 8 connection(s)
I20260620 09:17:03.482210 9017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:17:03.483350 9085 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:17:03.483981 9085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c: Bootstrap starting.
I20260620 09:17:03.491348 9085 log.cc:824] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:17:03.491789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 9017
I20260620 09:17:03.492077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40957
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44519
--webserver_port=35883
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:17:03.493291 9085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:17:03.493521 9085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c: Bootstrap complete.
I20260620 09:17:03.495636 9085 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44519 } }
I20260620 09:17:03.495873 9085 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9f1d7b081248fa98a31a13c3432a4c, State: Initialized, Role: FOLLOWER
I20260620 09:17:03.496034 9085 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44519 } }
I20260620 09:17:03.496106 9085 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:17:03.496151 9085 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:17:03.496196 9085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:17:03.497104 9085 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44519 } }
I20260620 09:17:03.497190 9085 leader_election.cc:302] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb9f1d7b081248fa98a31a13c3432a4c; no voters:
I20260620 09:17:03.497331 9085 leader_election.cc:288] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:17:03.497372 9089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:17:03.497483 9089 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [term 2 LEADER]: Becoming Leader. State: Replica: fb9f1d7b081248fa98a31a13c3432a4c, State: Running, Role: LEADER
I20260620 09:17:03.497574 9085 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:17:03.497560 9089 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44519 } }
I20260620 09:17:03.497802 9089 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb9f1d7b081248fa98a31a13c3432a4c. Latest consensus state: current_term: 2 leader_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44519 } } }
I20260620 09:17:03.497884 9089 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [sys.catalog]: This master's current role is: LEADER
I20260620 09:17:03.497807 9090 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9f1d7b081248fa98a31a13c3432a4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44519 } } }
I20260620 09:17:03.498026 9090 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c [sys.catalog]: This master's current role is: LEADER
I20260620 09:17:03.498232 9093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:17:03.498728 9093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:17:03.498852 9093 catalog_manager.cc:1259] Loaded cluster ID: 4eb59c23a5d34f14b5fc390f0c778f53
I20260620 09:17:03.498888 9093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:17:03.499920 9093 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:17:03.500090 9093 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb9f1d7b081248fa98a31a13c3432a4c: Loaded TSK: 0
I20260620 09:17:03.500279 9093 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:17:03.586968 9088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:03.587172 9088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:03.587193 9088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:03.588377 9088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:03.588416 9088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:03.588441 9088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:17:03.589794 9088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35883
--tserver_master_addrs=127.0.98.62:44519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:03.589993 9088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:03.590209 9088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:03.595727 9111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:03.595788 9112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:03.595820 9114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:03.596131 9088 server_base.cc:1034] running on GCE node
I20260620 09:17:03.596346 9088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:03.596616 9088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:03.597777 9088 hybrid_clock.cc:648] HybridClock initialized: now 1781947023597760 us; error 30 us; skew 500 ppm
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947023, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:17:03.600792 9088 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 09:17:03.601272 9088 webserver.cc:460] Webserver started at http://127.0.98.1:35883/ using document root <none> and password file <none>
I20260620 09:17:03.601513 9088 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:03.601586 9088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:03.603180 9088 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:17:03.603806 9121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.603960 9088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.604039 9088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "946a87e49e7d4656b6e2672fcd036afd"
format_stamp: "Formatted at 2026-06-20 09:17:02 on dist-test-slave-rq8c"
server_key: "8a9b47b41b33d1143c23b1fbdb6efd84"
server_key_iv: "73ada0759136752b7d90e01a4a547e36"
server_key_version: "encryptionkey@0"
I20260620 09:17:03.604375 9088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:03.624679 9088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:03.626732 9088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:03.626868 9088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:03.627110 9088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:03.627410 9088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:03.627444 9088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.627473 9088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:03.627496 9088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.633076 9088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40957
I20260620 09:17:03.633127 9234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40957 every 8 connection(s)
I20260620 09:17:03.633481 9088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:17:03.636971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 9088
I20260620 09:17:03.637230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46515
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44519
--webserver_port=36777
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:03.642201 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.634189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:46231)
0620 09:17:03.634350 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:03.634392 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:03.634851 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:03.635264 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:03.635267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:03.635881 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:03.636072 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.636078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.637628 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.637630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.638719 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.638723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.639282 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.639283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.639404 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.639948 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:03.639957 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:03.641996 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
W20260620 09:17:03.642277 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:03.634250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46231 (local address 127.0.98.62:44519)
0620 09:17:03.634444 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:03.634453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:03.634486 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:17:03.635022 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:03.635028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:03.635088 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:03.635224 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:03.636167 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.637505 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.638802 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.639204 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.639299 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.642133 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46231: BlockingRecv error: recv got EOF from 127.0.98.1:46231 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
W20260620 09:17:03.642427 9235 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44519 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44519: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:03.647018 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.642626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:45453)
0620 09:17:03.642746 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:03.642758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:03.642815 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:03.642943 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:03.642944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:03.643086 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:03.643144 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.643145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.643770 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.643771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.644566 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.644569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.644900 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.644901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.644983 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.645366 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:03.645371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:03.646957 (+ 1586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102}
W20260620 09:17:03.647092 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:03.642698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45453 (local address 127.0.98.62:44519)
0620 09:17:03.642743 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:03.642744 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:03.642806 (+ 62us) server_negotiation.cc:407] Connection header received
0620 09:17:03.642866 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:03.642867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:03.642876 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:03.642916 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:03.643206 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.643708 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.644637 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.644810 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.644852 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.647042 (+ 2190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45453: BlockingRecv error: recv got EOF from 127.0.98.1:45453 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:03.650799 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.647252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:50167)
0620 09:17:03.647288 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:03.647297 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:03.647328 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:03.647438 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:03.647439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:03.647543 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:03.647568 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.647569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.648145 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.648145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.648612 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.648613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.648903 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.648904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.648949 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.649283 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:03.649287 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:03.650729 (+ 1442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 09:17:03.650842 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:03.647297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50167 (local address 127.0.98.62:44519)
0620 09:17:03.647325 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:03.647327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:03.647334 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:17:03.647381 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:03.647381 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:03.647390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:03.647433 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:03.647644 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.648083 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.648675 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.648851 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.648901 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.650804 (+ 1903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50167: BlockingRecv error: recv got EOF from 127.0.98.1:50167 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260620 09:17:03.650913 9235 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:17:03.717443 9239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:03.717635 9239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:03.717657 9239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:03.718727 9239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:03.718760 9239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:03.718784 9239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:17:03.720155 9239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36777
--tserver_master_addrs=127.0.98.62:44519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:03.720329 9239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:03.720495 9239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:03.725055 9245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:03.725028 9247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:03.725095 9239 server_base.cc:1034] running on GCE node
W20260620 09:17:03.725004 9244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:03.725406 9239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:03.725646 9239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:03.726773 9239 hybrid_clock.cc:648] HybridClock initialized: now 1781947023726764 us; error 26 us; skew 500 ppm
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947023, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:17:03.729061 9239 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 09:17:03.729429 9239 webserver.cc:460] Webserver started at http://127.0.98.2:36777/ using document root <none> and password file <none>
I20260620 09:17:03.729600 9239 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:03.729648 9239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:03.730811 9239 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:17:03.731307 9254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.731455 9239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.731544 9239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "8a5a91470e494c349cf15a27b5efac9a"
format_stamp: "Formatted at 2026-06-20 09:17:03 on dist-test-slave-rq8c"
server_key: "d613da93cd53e530b31d82a93f496196"
server_key_iv: "3f8b2eb637e20dde6de8493be5442c0e"
server_key_version: "encryptionkey@0"
I20260620 09:17:03.731909 9239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:03.759025 9239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:03.761267 9239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:03.761407 9239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:03.761636 9239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:03.761943 9239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:03.761989 9239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.762019 9239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:03.762043 9239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.767436 9239 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46515
I20260620 09:17:03.767549 9367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46515 every 8 connection(s)
I20260620 09:17:03.767833 9239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:17:03.772253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 9239
I20260620 09:17:03.772526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33371
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44519
--webserver_port=43669
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:17:03.774837 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:03.768496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40409 (local address 127.0.98.62:44519)
0620 09:17:03.768547 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:03.768550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:03.768769 (+ 219us) server_negotiation.cc:407] Connection header received
0620 09:17:03.769240 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:03.769241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:03.769254 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:03.769304 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:03.770157 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.770635 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.771571 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.771781 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.771823 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.774789 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40409: BlockingRecv error: recv got EOF from 127.0.98.2:40409 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:17:03.774900 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.768502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:40409)
0620 09:17:03.768656 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:03.768689 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:03.769167 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:03.769358 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:03.769362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:03.769890 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:03.770092 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.770098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.770693 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.770694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.771507 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.771509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.771834 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.771835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.771937 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.772469 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:03.772477 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:03.774705 (+ 2228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
W20260620 09:17:03.775127 9368 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44519 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44519: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:03.779695 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.775357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:47853)
0620 09:17:03.775450 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:03.775460 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:03.775506 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:03.775660 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:03.775661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:03.775796 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:03.775851 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.775853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.776451 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.776452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.777261 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.777263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.777555 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.777556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.777619 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.777973 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:03.777978 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:03.779605 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 09:17:03.779783 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:03.775412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47853 (local address 127.0.98.62:44519)
0620 09:17:03.775460 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:03.775462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:03.775492 (+ 30us) server_negotiation.cc:407] Connection header received
0620 09:17:03.775555 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:03.775556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:03.775564 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:03.775632 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:03.775939 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.776386 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.777332 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.777499 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.777544 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.779725 (+ 2181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47853: BlockingRecv error: recv got EOF from 127.0.98.2:47853 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:03.783958 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.779967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:40363)
0620 09:17:03.780008 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:03.780017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:03.780050 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:03.780177 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:03.780178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:03.780275 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:03.780301 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.780302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.781140 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.781140 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.781603 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.781604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.782024 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.782025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.782056 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.782387 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:03.782392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:03.783885 (+ 1493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 09:17:03.783990 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:03.780037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40363 (local address 127.0.98.62:44519)
0620 09:17:03.780074 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:03.780076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:03.780083 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:17:03.780102 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:03.780103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:03.780111 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:03.780174 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:03.780361 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.781072 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.781671 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.781961 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.782030 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.783949 (+ 1919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40363: BlockingRecv error: recv got EOF from 127.0.98.2:40363 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 09:17:03.784178 9368 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:17:03.852197 9371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:03.852375 9371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:03.852396 9371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:03.853458 9371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:03.853490 9371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:03.853514 9371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:17:03.854842 9371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43669
--tserver_master_addrs=127.0.98.62:44519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:03.855015 9371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:03.855178 9371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:03.859844 9379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:03.859844 9376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:03.859989 9371 server_base.cc:1034] running on GCE node
W20260620 09:17:03.859874 9377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:03.860226 9371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:03.860422 9371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:03.861546 9371 hybrid_clock.cc:648] HybridClock initialized: now 1781947023861531 us; error 32 us; skew 500 ppm
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947023, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:17:03.863897 9371 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 09:17:03.864293 9371 webserver.cc:460] Webserver started at http://127.0.98.3:43669/ using document root <none> and password file <none>
I20260620 09:17:03.864475 9371 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:03.864537 9371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:03.865772 9371 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:17:03.866245 9386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.866386 9371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.866462 9371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "740360835dc0467d9c3ae6c351007cd9"
format_stamp: "Formatted at 2026-06-20 09:17:03 on dist-test-slave-rq8c"
server_key: "c19f5e5a07f51b6a7bbb9b8443fbf5a5"
server_key_iv: "246b7e4c0450979b3fa8bb4d5b0d9f21"
server_key_version: "encryptionkey@0"
I20260620 09:17:03.866802 9371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:03.884248 9371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:03.886313 9371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:03.886442 9371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:03.886658 9371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:03.886957 9371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:03.886999 9371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.887027 9371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:03.887048 9371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:03.892246 9371 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33371
I20260620 09:17:03.892299 9499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33371 every 8 connection(s)
I20260620 09:17:03.892604 9371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:17:03.897197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 9371
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:17:03.899729 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:03.893248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49399 (local address 127.0.98.62:44519)
0620 09:17:03.893300 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:03.893302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:03.893498 (+ 196us) server_negotiation.cc:407] Connection header received
0620 09:17:03.893980 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:03.893981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:03.893993 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:03.894042 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:03.894948 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.895421 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.896408 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.896610 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.896665 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.899673 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49399: BlockingRecv error: recv got EOF from 127.0.98.3:49399 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:17:03.899792 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.893245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:49399)
0620 09:17:03.893388 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:03.893423 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:03.893893 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:03.894124 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:03.894128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:03.894675 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:03.894858 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.894863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.895497 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.895498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.896331 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.896334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.896676 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.896677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.896803 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.897303 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:03.897313 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:03.899572 (+ 2259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
W20260620 09:17:03.900091 9500 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44519 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44519: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:03.904253 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.897651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.0.98.62:44519)
0620 09:17:03.897755 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:03.897757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:03.897764 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:17:03.897783 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:03.897784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:03.897792 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:03.897842 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:03.898548 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.899050 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.899915 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.900132 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.900197 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.901416 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:03.901420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:03.901420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:17:03.901430 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:03.902923 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:03.903287 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:03.903291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:03.903291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:03.903405 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:03.903572 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:03.903572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:03.903573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:03.903822 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:03.903962 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:03.904054 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 09:17:03.904157 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":49,"threads_started":1}
I20260620 09:17:03.904583 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.900293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:58771)
0620 09:17:03.900334 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:03.900343 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:03.900378 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:03.900474 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:03.900475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:03.900616 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:03.900660 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.900661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.901349 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.901350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.902015 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.902018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.902433 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.902434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.902484 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.902798 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:03.902803 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:03.904533 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 09:17:03.904652 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:03.900335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58771 (local address 127.0.98.62:44519)
0620 09:17:03.900373 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:03.900375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:03.900382 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:17:03.900423 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:03.900424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:03.900430 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:03.900469 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:03.900725 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.901178 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.902089 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.902351 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.902428 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.904604 (+ 2176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58771: BlockingRecv error: recv got EOF from 127.0.98.3:58771 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:03 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:03.908356 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:03.904772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:56781)
0620 09:17:03.904894 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:03.904903 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:03.904957 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:03.905094 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:03.905095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:03.905224 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:03.905267 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.905268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.905784 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.905784 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.906309 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:03.906310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.906627 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.906628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:03.906670 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.906973 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:03.906976 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:03.908305 (+ 1329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
W20260620 09:17:03.908418 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:03.904868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56781 (local address 127.0.98.62:44519)
0620 09:17:03.904899 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:03.904900 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:03.904946 (+ 46us) server_negotiation.cc:407] Connection header received
0620 09:17:03.904997 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:03.904998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:03.905005 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:03.905054 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:03.905317 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.905734 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.906369 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:03.906559 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:03.906603 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:03.908372 (+ 1769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56781: BlockingRecv error: recv got EOF from 127.0.98.3:56781 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 09:17:03.908478 9500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 09:17:04 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:04 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:04.657953 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:04.651284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:36017)
0620 09:17:04.651412 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:04.651422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:04.651467 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:04.651603 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:04.651604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:04.651776 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:04.651826 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:04.651828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:04.652432 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:04.652432 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:04.652986 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:04.652988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:04.653272 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:04.653273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:04.653308 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:04.656172 (+ 2864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:04.656177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:04.657876 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
W20260620 09:17:04.658036 9507 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:04.651375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36017 (local address 127.0.98.62:44519)
0620 09:17:04.651484 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:04.651485 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:04.651493 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:17:04.651514 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:04.651514 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:04.651523 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:04.651584 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:04.651887 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:04.652376 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:04.653044 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:04.653226 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:04.653268 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:04.657967 (+ 4699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36017: BlockingRecv error: recv got EOF from 127.0.98.1:36017 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
Jun 20 09:17:04 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:04 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:04.788789 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:04.784605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:47499)
0620 09:17:04.784711 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:04.784723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:04.784785 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:04.784910 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:04.784911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:04.785093 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:04.785140 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:04.785142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:04.785746 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:04.785747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:04.786286 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:04.786288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:04.786587 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:04.786587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:04.786628 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:04.787112 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:04.787117 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:04.788720 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":61,"threads_started":1}
W20260620 09:17:04.788862 9507 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:04.784627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47499 (local address 127.0.98.62:44519)
0620 09:17:04.784699 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:04.784701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:04.784769 (+ 68us) server_negotiation.cc:407] Connection header received
0620 09:17:04.784834 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:04.784835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:04.784846 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:04.784902 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:04.785202 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:04.785689 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:04.786357 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:04.786534 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:04.786580 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:04.788806 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47499: BlockingRecv error: recv got EOF from 127.0.98.2:47499 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 09:17:04 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:04 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:04.912986 9509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:04.908836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:35069)
0620 09:17:04.908950 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:04.908963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:04.909033 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:04.909154 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:04.909155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:04.909316 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:04.909372 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:04.909374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:04.909996 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:04.909997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:04.910514 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:04.910516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:04.910839 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:04.910840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:04.910910 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:04.911317 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:04.911321 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:04.912910 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
W20260620 09:17:04.913056 9507 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:04.908927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35069 (local address 127.0.98.62:44519)
0620 09:17:04.908973 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:04.908975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:04.909029 (+ 54us) server_negotiation.cc:407] Connection header received
0620 09:17:04.909083 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:04.909084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:04.909097 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:04.909144 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:04.909445 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:04.909926 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:04.910594 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:04.910776 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:04.910831 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:04.913001 (+ 2170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35069: BlockingRecv error: recv got EOF from 127.0.98.3:35069 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 09:17:05 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:05 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:05.662870 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:05.658368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:33377)
0620 09:17:05.658517 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:05.658532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:05.658588 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:05.658727 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:05.658728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:05.658878 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:05.658926 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:05.658928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:05.659536 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:05.659537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:05.660154 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:05.660156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:05.660592 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:05.660593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:05.660643 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:05.661076 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:05.661081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:05.662810 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
W20260620 09:17:05.662971 9511 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:05.658478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33377 (local address 127.0.98.62:44519)
0620 09:17:05.658582 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:05.658583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:05.658597 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:17:05.658622 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:05.658622 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:05.658632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:05.658689 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:05.658991 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:05.659479 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:05.660224 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:05.660493 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:05.660546 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:05.662900 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33377: BlockingRecv error: recv got EOF from 127.0.98.1:33377 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
Jun 20 09:17:05 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:05 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:05.793534 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:05.789212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:55687)
0620 09:17:05.789368 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:05.789390 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:05.789443 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:05.789563 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:05.789564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:05.789717 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:05.789767 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:05.789769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:05.790373 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:05.790374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:05.790930 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:05.790932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:05.791287 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:05.791288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:05.791323 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:05.791814 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:05.791819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:05.793456 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
W20260620 09:17:05.793622 9511 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:05.789288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55687 (local address 127.0.98.62:44519)
0620 09:17:05.789343 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:05.789344 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:05.789434 (+ 90us) server_negotiation.cc:407] Connection header received
0620 09:17:05.789486 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:05.789487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:05.789498 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:05.789550 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:05.789817 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:05.790316 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:05.791020 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:05.791225 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:05.791286 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:05.793558 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55687: BlockingRecv error: recv got EOF from 127.0.98.2:55687 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 09:17:05 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:05 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:05.917728 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:05.913469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:39731)
0620 09:17:05.913591 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:05.913610 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:05.913674 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:05.913803 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:05.913805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:05.913994 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:05.914043 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:05.914044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:05.914679 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:05.914680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:05.915209 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:05.915211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:05.915524 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:05.915525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:05.915579 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:05.916045 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:05.916050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:05.917651 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
W20260620 09:17:05.917802 9511 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:05.913557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39731 (local address 127.0.98.62:44519)
0620 09:17:05.913598 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:05.913600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:05.913671 (+ 71us) server_negotiation.cc:407] Connection header received
0620 09:17:05.913729 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:05.913730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:05.913745 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:05.913790 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:05.914113 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:05.914606 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:05.915278 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:05.915464 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:05.915520 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:05.917748 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39731: BlockingRecv error: recv got EOF from 127.0.98.3:39731 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 09:17:06 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:06 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:06.667745 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:06.663293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:57419)
0620 09:17:06.663422 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:06.663432 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:06.663486 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:06.663601 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:06.663602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:06.663773 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:06.663822 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:06.663824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:06.664524 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:06.664525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:06.665061 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:06.665063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:06.665396 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:06.665398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:06.665486 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:06.665981 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:06.665985 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:06.667676 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
W20260620 09:17:06.667845 9515 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:06.663354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57419 (local address 127.0.98.62:44519)
0620 09:17:06.663463 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:06.663465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:06.663492 (+ 27us) server_negotiation.cc:407] Connection header received
0620 09:17:06.663535 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:06.663536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:06.663546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:06.663593 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:06.663900 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:06.664446 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:06.665113 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:06.665317 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:06.665372 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:06.667757 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57419: BlockingRecv error: recv got EOF from 127.0.98.1:57419 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
Jun 20 09:17:06 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:06 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:06.798218 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:06.794041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:42377)
0620 09:17:06.794153 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:06.794164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:06.794213 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:06.794348 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:06.794348 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:06.794536 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:06.794585 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:06.794587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:06.795187 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:06.795188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:06.795781 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:06.795783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:06.796057 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:06.796058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:06.796095 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:06.796514 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:06.796519 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:06.798153 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
W20260620 09:17:06.798345 9515 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:06.794121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42377 (local address 127.0.98.62:44519)
0620 09:17:06.794221 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:06.794223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:06.794232 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:17:06.794257 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:06.794258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:06.794268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:06.794332 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:06.794647 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:06.795129 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:06.795838 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:06.796010 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:06.796053 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:06.798242 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42377: BlockingRecv error: recv got EOF from 127.0.98.2:42377 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 09:17:06 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:06 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:06.922495 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:06.918230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:49373)
0620 09:17:06.918375 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:06.918385 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:06.918441 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:06.918562 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:06.918563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:06.918731 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:06.918780 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:06.918782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:06.919413 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:06.919414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:06.919989 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:06.919990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:06.920306 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:06.920307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:06.920361 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:06.920813 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:06.920817 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:06.922437 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
W20260620 09:17:06.922628 9515 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:06.918314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49373 (local address 127.0.98.62:44519)
0620 09:17:06.918388 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:06.918390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:06.918427 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:17:06.918489 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:06.918490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:06.918502 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:06.918551 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:06.918856 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:06.919348 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:06.920047 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:06.920225 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:06.920281 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:06.922556 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49373: BlockingRecv error: recv got EOF from 127.0.98.3:49373 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 09:17:07 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:07 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:07.672552 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:07.668167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:37955)
0620 09:17:07.668322 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:07.668332 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:07.668416 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:07.668561 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:07.668562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:07.668759 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:07.668821 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:07.668824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:07.669552 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:07.669553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:07.670078 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:07.670079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:07.670479 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:07.670480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:07.670521 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:07.670945 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:07.670949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:07.672495 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
W20260620 09:17:07.672652 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:07.668233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37955 (local address 127.0.98.62:44519)
0620 09:17:07.668336 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:07.668338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:07.668393 (+ 55us) server_negotiation.cc:407] Connection header received
0620 09:17:07.668470 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:07.668471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:07.668481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:07.668555 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:07.668892 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:07.669493 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:07.670168 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:07.670419 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:07.670478 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:07.672579 (+ 2101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37955: BlockingRecv error: recv got EOF from 127.0.98.1:37955 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":46,"threads_started":1}
Jun 20 09:17:07 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:07 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:07.802892 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:07.798672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:44365)
0620 09:17:07.798785 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:07.798796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:07.798865 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:07.799001 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:07.799001 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:07.799175 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:07.799221 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:07.799223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:07.799930 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:07.799932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:07.800452 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:07.800454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:07.800776 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:07.800777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:07.800831 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:07.801273 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:07.801278 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:07.802816 (+ 1538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
W20260620 09:17:07.802963 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:07.798698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44365 (local address 127.0.98.62:44519)
0620 09:17:07.798777 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:07.798779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:07.798856 (+ 77us) server_negotiation.cc:407] Connection header received
0620 09:17:07.798930 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:07.798931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:07.798942 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:07.798991 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:07.799295 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:07.799864 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:07.800526 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:07.800718 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:07.800771 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:07.802910 (+ 2139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44365: BlockingRecv error: recv got EOF from 127.0.98.2:44365 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 09:17:07 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:07 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:07.927263 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:07.922954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:38191)
0620 09:17:07.923106 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:07.923117 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:07.923189 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:07.923321 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:07.923321 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:07.923519 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:07.923580 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:07.923583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:07.924252 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:07.924252 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:07.924792 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:07.924793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:07.925063 (+ 270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:07.925064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:07.925101 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:07.925492 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:07.925496 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:07.927187 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
W20260620 09:17:07.927345 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:07.923027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38191 (local address 127.0.98.62:44519)
0620 09:17:07.923092 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:07.923094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:07.923188 (+ 94us) server_negotiation.cc:407] Connection header received
0620 09:17:07.923247 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:07.923248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:07.923262 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:07.923312 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:07.923678 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:07.924182 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:07.924847 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:07.925018 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:07.925058 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:07.927291 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38191: BlockingRecv error: recv got EOF from 127.0.98.3:38191 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 09:17:08 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:08 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:08.677333 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:08.672988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:39523)
0620 09:17:08.673147 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:08.673159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:08.673227 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:08.673353 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:08.673353 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:08.673505 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:08.673554 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:08.673556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:08.674161 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:08.674161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:08.674781 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:08.674785 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:08.675163 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:08.675171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:08.675226 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:08.675671 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:08.675677 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:08.677247 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
W20260620 09:17:08.677400 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:08.673068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39523 (local address 127.0.98.62:44519)
0620 09:17:08.673197 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:08.673199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:08.673213 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:17:08.673278 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:08.673279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:08.673288 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:08.673342 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:08.673603 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:08.674103 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:08.674851 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:08.675103 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:08.675158 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:08.677340 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39523: BlockingRecv error: recv got EOF from 127.0.98.1:39523 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
Jun 20 09:17:08 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:08 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:08.807866 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:08.803338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:46871)
0620 09:17:08.803460 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:08.803472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:08.803533 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:08.803719 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:08.803720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:08.803910 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:08.803958 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:08.803960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:08.804580 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:08.804581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:08.805128 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:08.805130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:08.805459 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:08.805460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:08.805517 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:08.805972 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:08.805977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:08.807807 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
W20260620 09:17:08.807955 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:08.803437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46871 (local address 127.0.98.62:44519)
0620 09:17:08.803487 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:08.803489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:08.803518 (+ 29us) server_negotiation.cc:407] Connection header received
0620 09:17:08.803595 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:08.803596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:08.803607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:08.803711 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:08.804017 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:08.804513 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:08.805190 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:08.805392 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:08.805448 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:08.807894 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46871: BlockingRecv error: recv got EOF from 127.0.98.2:46871 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 09:17:08 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:08 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:08.931870 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:08.927735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:45265)
0620 09:17:08.927856 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:08.927866 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:08.927930 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:08.928068 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:08.928069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:08.928228 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:08.928277 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:08.928278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:08.928920 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:08.928921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:08.929465 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:08.929466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:08.929742 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:08.929742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:08.929777 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:08.930201 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:08.930204 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:08.931780 (+ 1576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
W20260620 09:17:08.931938 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:08.927825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45265 (local address 127.0.98.62:44519)
0620 09:17:08.927868 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:08.927870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:08.927951 (+ 81us) server_negotiation.cc:407] Connection header received
0620 09:17:08.927979 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:08.927980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:08.927994 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:08.928051 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:08.928353 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:08.928862 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:08.929518 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:08.929697 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:08.929738 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:08.931873 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45265: BlockingRecv error: recv got EOF from 127.0.98.3:45265 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 09:17:09 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:09 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:09.685290 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:09.677806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:60715)
0620 09:17:09.677939 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:09.677958 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:09.678017 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:09.678145 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:09.678145 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:09.678306 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:09.678369 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:09.678371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:09.679077 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:09.679078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:09.679610 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:09.679612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:09.679995 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:09.679996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:09.680060 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:09.683347 (+ 3287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:09.683352 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:09.685209 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
W20260620 09:17:09.685384 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:09.677857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60715 (local address 127.0.98.62:44519)
0620 09:17:09.677968 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:09.677970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:09.678002 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:17:09.678056 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:09.678056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:09.678066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:09.678134 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:09.678471 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:09.678981 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:09.679712 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:09.679909 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:09.679958 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:09.685318 (+ 5360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60715: BlockingRecv error: recv got EOF from 127.0.98.1:60715 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
Jun 20 09:17:09 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:09 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:09.812466 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:09.808293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:58363)
0620 09:17:09.808436 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:09.808447 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:09.808516 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:09.808631 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:09.808632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:09.808787 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:09.808834 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:09.808836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:09.809422 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:09.809423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:09.809973 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:09.809975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:09.810254 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:09.810255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:09.810292 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:09.810715 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:09.810719 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:09.812398 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
W20260620 09:17:09.812541 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:09.808349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58363 (local address 127.0.98.62:44519)
0620 09:17:09.808400 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:09.808402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:09.808516 (+ 114us) server_negotiation.cc:407] Connection header received
0620 09:17:09.808562 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:09.808563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:09.808573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:09.808620 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:09.808886 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:09.809372 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:09.810030 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:09.810208 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:09.810250 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:09.812486 (+ 2236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58363: BlockingRecv error: recv got EOF from 127.0.98.2:58363 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 09:17:09 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:09 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:09.936977 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:09.932475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:48195)
0620 09:17:09.932613 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:09.932623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:09.932683 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:09.932804 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:09.932804 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:09.932980 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:09.933044 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:09.933046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:09.933723 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:09.933724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:09.934346 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:09.934349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:09.934688 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:09.934689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:09.934733 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:09.935166 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:09.935171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:09.936895 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
W20260620 09:17:09.937052 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:09.932546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48195 (local address 127.0.98.62:44519)
0620 09:17:09.932620 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:09.932621 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:09.932677 (+ 56us) server_negotiation.cc:407] Connection header received
0620 09:17:09.932724 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:09.932725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:09.932737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:09.932792 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:09.933116 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:09.933633 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:09.934450 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:09.934634 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:09.934685 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:09.936987 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48195: BlockingRecv error: recv got EOF from 127.0.98.3:48195 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 09:17:10 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:10 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:10.690244 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:10.685800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:43937)
0620 09:17:10.685970 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:10.685981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:10.686038 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:10.686155 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:10.686155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:10.686303 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:10.686350 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:10.686352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:10.686940 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:10.686940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:10.687463 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:10.687465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:10.687835 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:10.687835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:10.687887 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:10.688270 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:10.688274 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:10.690183 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
W20260620 09:17:10.690332 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:10.685870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43937 (local address 127.0.98.62:44519)
0620 09:17:10.685989 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:10.685990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:10.686028 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:17:10.686088 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:10.686088 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:10.686098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:10.686143 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:10.686399 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:10.686883 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:10.687554 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:10.687772 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:10.687827 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:10.690259 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43937: BlockingRecv error: recv got EOF from 127.0.98.1:43937 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
Jun 20 09:17:10 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:10 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:10.818266 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:10.812971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:43533)
0620 09:17:10.813109 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:10.813122 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:10.813195 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:10.813327 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:10.813328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:10.813492 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:10.813552 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:10.813554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:10.814195 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:10.814196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:10.814823 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:10.814828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:10.815529 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:10.815532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:10.815676 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:10.816342 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:10.816348 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:10.818188 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
W20260620 09:17:10.818359 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:10.813061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43533 (local address 127.0.98.62:44519)
0620 09:17:10.813127 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:10.813129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:10.813183 (+ 54us) server_negotiation.cc:407] Connection header received
0620 09:17:10.813254 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:10.813255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:10.813267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:10.813313 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:10.813623 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:10.814125 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:10.814935 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:10.815138 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:10.815517 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:10.818306 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43533: BlockingRecv error: recv got EOF from 127.0.98.2:43533 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 09:17:10 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:10 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:10.941753 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:10.937417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:41361)
0620 09:17:10.937538 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:10.937551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:10.937623 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:10.937742 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:10.937743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:10.937908 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:10.937953 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:10.937955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:10.938578 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:10.938579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:10.939103 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:10.939104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:10.939411 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:10.939411 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:10.939462 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:10.939916 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:10.939920 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:10.941695 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
W20260620 09:17:10.941869 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:10.937502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41361 (local address 127.0.98.62:44519)
0620 09:17:10.937561 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:10.937562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:10.937603 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:17:10.937670 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:10.937671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:10.937685 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:10.937732 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:10.938027 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:10.938512 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:10.939174 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:10.939355 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:10.939408 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:10.941791 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41361: BlockingRecv error: recv got EOF from 127.0.98.3:41361 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 09:17:11 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:11 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:11.698045 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:11.690760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:47813)
0620 09:17:11.690903 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:11.690912 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:11.690970 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:11.691132 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:11.691133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:11.691284 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:11.691331 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:11.691333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:11.692108 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:11.692110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:11.692679 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:11.692682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:11.693041 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:11.693042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:11.693097 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:11.696137 (+ 3040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:11.696143 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:11.697929 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
W20260620 09:17:11.698133 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:11.690906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47813 (local address 127.0.98.62:44519)
0620 09:17:11.691012 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:11.691014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:11.691021 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:17:11.691043 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:11.691043 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:11.691053 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:11.691118 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:11.691381 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:11.692039 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:11.692760 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:11.692972 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:11.693026 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:11.698059 (+ 5033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47813: BlockingRecv error: recv got EOF from 127.0.98.1:47813 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
Jun 20 09:17:11 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:11 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:11.823726 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:11.818798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:49939)
0620 09:17:11.818956 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:11.818966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:11.819016 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:11.819200 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:11.819201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:11.819375 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:11.819423 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:11.819425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:11.820522 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:11.820523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:11.821064 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:11.821066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:11.821408 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:11.821409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:11.821445 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:11.821956 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:11.821961 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:11.823666 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
W20260620 09:17:11.823818 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:11.818909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49939 (local address 127.0.98.62:44519)
0620 09:17:11.818990 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:11.818991 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:11.819006 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:17:11.819072 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:11.819073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:11.819084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:11.819157 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:11.819511 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:11.820439 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:11.821155 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:11.821353 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:11.821407 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:11.823746 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49939: BlockingRecv error: recv got EOF from 127.0.98.2:49939 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 09:17:11 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:11 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:11.946211 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:11.942199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:36981)
0620 09:17:11.942337 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:11.942347 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:11.942406 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:11.942528 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:11.942528 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:11.942678 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:11.942728 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:11.942729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:11.943381 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:11.943381 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:11.943929 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:11.943931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:11.944235 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:11.944235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:11.944270 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:11.944678 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:11.944682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:11.946155 (+ 1473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
W20260620 09:17:11.946295 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:11.942260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36981 (local address 127.0.98.62:44519)
0620 09:17:11.942318 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:11.942319 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:11.942403 (+ 84us) server_negotiation.cc:407] Connection header received
0620 09:17:11.942464 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:11.942465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:11.942476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:11.942513 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:11.942823 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:11.943322 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:11.943991 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:11.944182 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:11.944234 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:11.946234 (+ 2000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36981: BlockingRecv error: recv got EOF from 127.0.98.3:36981 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 09:17:12 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:12 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:12.702606 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:12.698439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:44789)
0620 09:17:12.698547 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:12.698558 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:12.698617 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:12.698757 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:12.698758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:12.698943 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:12.698990 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:12.698993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:12.699669 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:12.699670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:12.700186 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:12.700188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:12.700506 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:12.700506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:12.700541 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:12.700932 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:12.700935 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:12.702532 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
W20260620 09:17:12.702678 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:12.698491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44789 (local address 127.0.98.62:44519)
0620 09:17:12.698599 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:12.698601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:12.698610 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:17:12.698666 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:12.698667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:12.698678 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:12.698740 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:12.699050 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:12.699583 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:12.700255 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:12.700455 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:12.700503 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:12.702621 (+ 2118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44789: BlockingRecv error: recv got EOF from 127.0.98.1:44789 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
Jun 20 09:17:12 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:12 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:12.828547 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:12.824174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:47423)
0620 09:17:12.824285 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:12.824299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:12.824370 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:12.824503 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:12.824504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:12.824671 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:12.824715 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:12.824716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:12.825371 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:12.825372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:12.825895 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:12.825897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:12.826204 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:12.826205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:12.826252 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:12.826700 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:12.826705 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:12.828491 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":60,"threads_started":1}
W20260620 09:17:12.828612 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:12.824247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47423 (local address 127.0.98.62:44519)
0620 09:17:12.824310 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:12.824312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:12.824350 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:17:12.824428 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:12.824429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:12.824441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:12.824488 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:12.824795 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:12.825301 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:12.825967 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:12.826149 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:12.826202 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:12.828556 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47423: BlockingRecv error: recv got EOF from 127.0.98.2:47423 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 09:17:12 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:12 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:12.951092 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:12.946716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:37559)
0620 09:17:12.946844 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:12.946856 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:12.946930 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:12.947063 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:12.947064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:12.947274 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:12.947322 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:12.947325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:12.948004 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:12.948006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:12.948537 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:12.948538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:12.948855 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:12.948856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:12.948910 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:12.949345 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:12.949350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:12.951035 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
W20260620 09:17:12.951182 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:12.946815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37559 (local address 127.0.98.62:44519)
0620 09:17:12.946918 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:12.946920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:12.946930 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:17:12.946985 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:12.946986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:12.946999 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:12.947050 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:12.947399 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:12.947936 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:12.948613 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:12.948800 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:12.948850 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:12.951117 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37559: BlockingRecv error: recv got EOF from 127.0.98.3:37559 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 09:17:13 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:13 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:13.707603 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:13.703082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:52817)
0620 09:17:13.703205 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:13.703217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:13.703267 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:13.703493 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:13.703494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:13.703675 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:13.703724 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:13.703726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:13.704329 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:13.704330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:13.704874 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:13.704876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:13.705281 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:13.705282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:13.705334 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:13.705792 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:13.705796 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:13.707520 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":86,"threads_started":1}
W20260620 09:17:13.707701 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:13.703119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52817 (local address 127.0.98.62:44519)
0620 09:17:13.703250 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:13.703252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:13.703261 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:17:13.703411 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:13.703412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:13.703423 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:13.703478 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:13.703774 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:13.704274 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:13.704959 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:13.705188 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:13.705256 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:13.707639 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52817: BlockingRecv error: recv got EOF from 127.0.98.1:52817 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
Jun 20 09:17:13 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:13 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:13.833372 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:13.829008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:33559)
0620 09:17:13.829137 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:13.829147 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:13.829192 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:13.829305 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:13.829306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:13.829465 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:13.829508 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:13.829510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:13.830177 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:13.830178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:13.830722 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:13.830724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:13.831039 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:13.831040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:13.831077 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:13.831482 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:13.831485 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:13.833296 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
W20260620 09:17:13.833431 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:13.829101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33559 (local address 127.0.98.62:44519)
0620 09:17:13.829175 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:13.829176 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:13.829185 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:17:13.829239 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:13.829239 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:13.829249 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:13.829300 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:13.829594 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:13.830091 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:13.830796 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:13.830981 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:13.831034 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:13.833378 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33559: BlockingRecv error: recv got EOF from 127.0.98.2:33559 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 09:17:13 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:13 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:13.955951 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:13.951497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:35443)
0620 09:17:13.951645 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:13.951656 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:13.951710 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:13.951824 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:13.951825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:13.951977 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:13.952025 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:13.952026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:13.952684 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:13.952685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:13.953235 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:13.953237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:13.953587 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:13.953588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:13.953640 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:13.954074 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:13.954078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:13.955892 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
W20260620 09:17:13.956048 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:13.951590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35443 (local address 127.0.98.62:44519)
0620 09:17:13.951661 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:13.951663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:13.951708 (+ 45us) server_negotiation.cc:407] Connection header received
0620 09:17:13.951759 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:13.951759 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:13.951772 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:13.951809 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:13.952089 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:13.952587 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:13.953324 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:13.953506 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:13.953558 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:13.955981 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35443: BlockingRecv error: recv got EOF from 127.0.98.3:35443 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 09:17:14 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:14 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:14.715287 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:14.708105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:52915)
0620 09:17:14.708262 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:14.708273 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:14.708332 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:14.708476 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:14.708477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:14.708641 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:14.708694 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:14.708696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:14.709354 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:14.709356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:14.709906 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:14.709908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:14.710267 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:14.710268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:14.710325 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:14.713511 (+ 3186us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:14.713515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:14.715209 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
W20260620 09:17:14.715373 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:14.708207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52915 (local address 127.0.98.62:44519)
0620 09:17:14.708325 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:14.708327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:14.708337 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:17:14.708393 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:14.708393 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:14.708403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:14.708457 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:14.708761 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:14.709275 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:14.709985 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:14.710189 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:14.710237 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:14.715312 (+ 5075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52915: BlockingRecv error: recv got EOF from 127.0.98.1:52915 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
Jun 20 09:17:14 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:14 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:14.838522 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:14.833984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:60993)
0620 09:17:14.834112 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:14.834138 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:14.834206 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:14.834376 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:14.834377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:14.834534 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:14.834600 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:14.834602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:14.835412 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:14.835413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:14.835987 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:14.835989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:14.836311 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:14.836312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:14.836347 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:14.836837 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:14.836842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:14.838463 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
W20260620 09:17:14.838678 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:14.834108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60993 (local address 127.0.98.62:44519)
0620 09:17:14.834165 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:14.834166 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:14.834181 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:17:14.834292 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:14.834293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:14.834305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:14.834365 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:14.834680 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:14.835343 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:14.836063 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:14.836257 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:14.836310 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:14.838603 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60993: BlockingRecv error: recv got EOF from 127.0.98.2:60993 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 09:17:14 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:14 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:14.960924 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:14.956365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:59311)
0620 09:17:14.956503 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:14.956513 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:14.956568 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:14.956748 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:14.956749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:14.956948 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:14.957006 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:14.957008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:14.957770 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:14.957772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:14.958333 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:14.958335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:14.958691 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:14.958692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:14.958737 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:14.959133 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:14.959138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:14.960846 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
W20260620 09:17:14.960985 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:14.956409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59311 (local address 127.0.98.62:44519)
0620 09:17:14.956481 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:14.956483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:14.956556 (+ 73us) server_negotiation.cc:407] Connection header received
0620 09:17:14.956646 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:14.956647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:14.956664 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:14.956741 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:14.957105 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:14.957681 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:14.958425 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:14.958615 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:14.958680 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:14.960933 (+ 2253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59311: BlockingRecv error: recv got EOF from 127.0.98.3:59311 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 09:17:15 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:15 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:15.720577 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:15.715860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:41429)
0620 09:17:15.715984 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:15.715995 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:15.716046 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:15.716223 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:15.716224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:15.716392 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:15.716442 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:15.716444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:15.717301 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:15.717303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:15.717916 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:15.717918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:15.718277 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:15.718278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:15.718332 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:15.718748 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:15.718752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:15.720487 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
W20260620 09:17:15.720664 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:15.715961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41429 (local address 127.0.98.62:44519)
0620 09:17:15.716093 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:15.716095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:15.716106 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:17:15.716129 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:15.716129 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:15.716139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:15.716212 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:15.716534 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:15.717217 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:15.718014 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:15.718215 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:15.718271 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:15.720599 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41429: BlockingRecv error: recv got EOF from 127.0.98.1:41429 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
Jun 20 09:17:15 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:15 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:15.843164 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:15.838960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:48881)
0620 09:17:15.839095 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:15.839107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:15.839161 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:15.839280 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:15.839280 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:15.839432 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:15.839487 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:15.839488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:15.840117 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:15.840118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:15.840669 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:15.840671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:15.840963 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:15.840964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:15.841009 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:15.841445 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:15.841455 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:15.843095 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
W20260620 09:17:15.843248 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:15.839044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48881 (local address 127.0.98.62:44519)
0620 09:17:15.839098 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:15.839100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:15.839157 (+ 57us) server_negotiation.cc:407] Connection header received
0620 09:17:15.839217 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:15.839218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:15.839228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:15.839269 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:15.839537 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:15.840039 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:15.840728 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:15.840903 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:15.840949 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:15.843184 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48881: BlockingRecv error: recv got EOF from 127.0.98.2:48881 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 09:17:15 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:15 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:15.965713 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:15.961395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:52789)
0620 09:17:15.961515 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:15.961527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:15.961588 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:15.961710 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:15.961711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:15.961871 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:15.961917 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:15.961918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:15.962500 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:15.962500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:15.963022 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:15.963023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:15.963351 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:15.963352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:15.963394 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:15.963832 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:15.963836 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:15.965636 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
W20260620 09:17:15.965787 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:15.961469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52789 (local address 127.0.98.62:44519)
0620 09:17:15.961556 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:15.961557 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:15.961571 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:17:15.961633 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:15.961634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:15.961655 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:15.961700 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:15.961968 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:15.962450 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:15.963093 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:15.963284 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:15.963355 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:15.965726 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52789: BlockingRecv error: recv got EOF from 127.0.98.3:52789 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 09:17:16 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:16 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:16.725114 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:16.721058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:45347)
0620 09:17:16.721185 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:16.721195 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:16.721245 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:16.721359 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:16.721359 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:16.721515 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:16.721562 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:16.721563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:16.722179 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:16.722179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:16.722691 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:16.722692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:16.723019 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:16.723020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:16.723057 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:16.723474 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:16.723478 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:16.725054 (+ 1576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
W20260620 09:17:16.725210 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:16.721136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45347 (local address 127.0.98.62:44519)
0620 09:17:16.721240 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:16.721242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:16.721250 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:17:16.721285 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:16.721286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:16.721295 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:16.721349 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:16.721622 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:16.722121 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:16.722762 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:16.722957 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:16.723010 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:16.725137 (+ 2127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45347: BlockingRecv error: recv got EOF from 127.0.98.1:45347 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":46,"threads_started":1}
Jun 20 09:17:16 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:16 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:16.848031 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:16.843596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:43651)
0620 09:17:16.843804 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:16.843815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:16.843887 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:16.844039 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:16.844040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:16.844212 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:16.844259 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:16.844260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:16.844874 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:16.844875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:16.845545 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:16.845548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:16.845873 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:16.845874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:16.845909 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:16.846363 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:16.846368 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:16.847973 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
W20260620 09:17:16.848161 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:16.843718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43651 (local address 127.0.98.62:44519)
0620 09:17:16.843778 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:16.843780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:16.843885 (+ 105us) server_negotiation.cc:407] Connection header received
0620 09:17:16.843962 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:16.843963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:16.843975 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:16.844035 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:16.844315 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:16.844800 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:16.845631 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:16.845816 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:16.845870 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:16.848092 (+ 2222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43651: BlockingRecv error: recv got EOF from 127.0.98.2:43651 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 09:17:16 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:16 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:16.970196 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:16.966148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:39091)
0620 09:17:16.966286 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:16.966296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:16.966356 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:16.966482 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:16.966483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:16.966635 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:16.966682 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:16.966683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:16.967269 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:16.967270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:16.967816 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:16.967817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:16.968115 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:16.968116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:16.968157 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:16.968555 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:16.968559 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:16.970132 (+ 1573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
W20260620 09:17:16.970263 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:16.966201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39091 (local address 127.0.98.62:44519)
0620 09:17:16.966262 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:16.966264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:16.966351 (+ 87us) server_negotiation.cc:407] Connection header received
0620 09:17:16.966402 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:16.966403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:16.966416 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:16.966466 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:16.966731 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:16.967215 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:16.967874 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:16.968063 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:16.968114 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:16.970207 (+ 2093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39091: BlockingRecv error: recv got EOF from 127.0.98.3:39091 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 09:17:17 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:17 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:17.729732 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:17.725579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:48787)
0620 09:17:17.725704 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:17.725714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:17.725761 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:17.725944 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:17.725945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:17.726105 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:17.726157 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:17.726159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:17.726784 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:17.726785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:17.727318 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:17.727320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:17.727700 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:17.727701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:17.727738 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:17.728130 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:17.728134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:17.729662 (+ 1528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
W20260620 09:17:17.729841 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:17.725677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48787 (local address 127.0.98.62:44519)
0620 09:17:17.725827 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:17.725829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:17.725837 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:17:17.725856 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:17.725857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:17.725866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:17.725934 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:17.726223 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:17.726727 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:17.727394 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:17.727594 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:17.727695 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:17.729765 (+ 2070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48787: BlockingRecv error: recv got EOF from 127.0.98.1:48787 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
Jun 20 09:17:17 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:17 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:17.852617 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:17.848488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:45109)
0620 09:17:17.848636 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:17.848646 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:17.848702 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:17.848813 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:17.848813 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:17.848974 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:17.849032 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:17.849033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:17.849620 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:17.849621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:17.850153 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:17.850155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:17.850509 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:17.850510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:17.850552 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:17.850975 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:17.850979 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:17.852557 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
W20260620 09:17:17.852679 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:17.848551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45109 (local address 127.0.98.62:44519)
0620 09:17:17.848606 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:17.848608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:17.848697 (+ 89us) server_negotiation.cc:407] Connection header received
0620 09:17:17.848754 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:17.848754 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:17.848764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:17.848801 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:17.849082 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:17.849568 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:17.850238 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:17.850439 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:17.850498 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:17.852621 (+ 2123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45109: BlockingRecv error: recv got EOF from 127.0.98.2:45109 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 09:17:17 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:17 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:17.974886 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:17.970626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.3:33023)
0620 09:17:17.970748 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:17.970759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:17.970826 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:17.970945 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:17.970946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:17.971101 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:17.971148 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:17.971150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:17.971860 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:17.971861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:17.972379 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:17.972381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:17.972707 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:17.972708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:17.972762 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:17.973233 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:17.973239 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:17.974826 (+ 1587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
W20260620 09:17:17.974984 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:17.970682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33023 (local address 127.0.98.62:44519)
0620 09:17:17.970751 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:17.970753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:17.970821 (+ 68us) server_negotiation.cc:407] Connection header received
0620 09:17:17.970871 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:17.970872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:17.970883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:17.970933 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:17.971246 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:17.971798 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:17.972444 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:17.972625 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:17.972677 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:17.974913 (+ 2236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33023: BlockingRecv error: recv got EOF from 127.0.98.3:33023 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 09:17:18 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:18 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:18.734295 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:18.730211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.1:60775)
0620 09:17:18.730323 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:18.730335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:18.730392 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:18.730508 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:18.730508 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:18.730653 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:18.730701 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:18.730702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:18.731303 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:18.731303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:18.731845 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:18.731847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:18.732133 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:18.732134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:18.732169 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:18.732562 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:18.732565 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:18.734229 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
W20260620 09:17:18.734378 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:18.730267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60775 (local address 127.0.98.62:44519)
0620 09:17:18.730404 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:18.730406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:18.730413 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:17:18.730434 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:18.730434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:18.730444 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:18.730498 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:18.730752 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:18.731252 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:18.731898 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:18.732084 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:18.732132 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:18.734310 (+ 2178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60775: BlockingRecv error: recv got EOF from 127.0.98.1:60775 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
Jun 20 09:17:18 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:17:18 dist-test-slave-rq8c krb5kdc[8504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:17:18.857358 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:18.853104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44519 (local address 127.0.98.2:48059)
0620 09:17:18.853218 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:18.853229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:18.853286 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:18.853391 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:18.853392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:18.853549 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:18.853593 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:18.853595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:18.854219 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:18.854219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:18.854731 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:18.854733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:18.855004 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:18.855005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:18.855042 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:18.855489 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:18.855494 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:18.857284 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44519: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
W20260620 09:17:18.857424 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:18.853156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48059 (local address 127.0.98.62:44519)
0620 09:17:18.853229 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:18.853231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:18.853272 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:17:18.853328 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:18.853328 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:18.853338 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:18.853382 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:18.853652 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:18.854165 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:18.854787 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:18.854957 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:18.855001 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:18.857370 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48059: BlockingRecv error: recv got EOF from 127.0.98.2:48059 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:17:18.904193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 9088
I20260620 09:17:18.908100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 9239
I20260620 09:17:18.911590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 9371
I20260620 09:17:18.914969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 9017
2026-06-20T09:17:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17292 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T09:17:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:17:18Z Disabled control of system clock
I20260620 09:17:18.940019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39413
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 09:17:19.016875 9568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:19.017066 9568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:19.017083 9568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:19.018169 9568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:17:19.018199 9568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:19.018210 9568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:19.018221 9568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:17:19.018236 9568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:17:19.019484 9568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:19.019701 9568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:19.019942 9568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:19.024556 9574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:19.024520 9576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.024569 9568 server_base.cc:1034] running on GCE node
W20260620 09:17:19.024520 9573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.024936 9568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:19.025184 9568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:19.026315 9568 hybrid_clock.cc:648] HybridClock initialized: now 1781947039026301 us; error 30 us; skew 500 ppm
I20260620 09:17:19.027396 9568 webserver.cc:460] Webserver started at http://127.0.98.62:40299/ using document root <none> and password file <none>
I20260620 09:17:19.027593 9568 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:19.027667 9568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:19.027799 9568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:19.028808 9568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "9be666d5838941b98ea1d1da887938c6"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "6acee002ef75f032a7f263884cbd07a0"
server_key_iv: "060731f7ecbed5ff45238416748af854"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.029135 9568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "9be666d5838941b98ea1d1da887938c6"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "6acee002ef75f032a7f263884cbd07a0"
server_key_iv: "060731f7ecbed5ff45238416748af854"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.030247 9568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:17:19.030829 9582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.030978 9568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:17:19.031055 9568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "9be666d5838941b98ea1d1da887938c6"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "6acee002ef75f032a7f263884cbd07a0"
server_key_iv: "060731f7ecbed5ff45238416748af854"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.031118 9568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:19.050582 9568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:19.050858 9568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:19.050990 9568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:19.054488 9568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39413
I20260620 09:17:19.054520 9634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39413 every 8 connection(s)
I20260620 09:17:19.054807 9568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:17:19.055426 9635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:17:19.057904 9635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6: Bootstrap starting.
I20260620 09:17:19.058437 9635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6: Neither blocks nor log segments found. Creating new log.
I20260620 09:17:19.058676 9635 log.cc:824] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6: Log is configured to *not* fsync() on all Append() calls
I20260620 09:17:19.059242 9635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6: No bootstrap required, opened a new log
I20260620 09:17:19.060765 9635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9be666d5838941b98ea1d1da887938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39413 } }
I20260620 09:17:19.060897 9635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:17:19.060930 9635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be666d5838941b98ea1d1da887938c6, State: Initialized, Role: FOLLOWER
I20260620 09:17:19.061065 9635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9be666d5838941b98ea1d1da887938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39413 } }
I20260620 09:17:19.061122 9635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:17:19.061143 9635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:17:19.061187 9635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:17:19.061645 9635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9be666d5838941b98ea1d1da887938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39413 } }
I20260620 09:17:19.061728 9635 leader_election.cc:302] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9be666d5838941b98ea1d1da887938c6; no voters:
I20260620 09:17:19.061883 9635 leader_election.cc:288] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:17:19.061914 9638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:17:19.062042 9638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [term 1 LEADER]: Becoming Leader. State: Replica: 9be666d5838941b98ea1d1da887938c6, State: Running, Role: LEADER
I20260620 09:17:19.062096 9635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:17:19.062155 9638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9be666d5838941b98ea1d1da887938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39413 } }
I20260620 09:17:19.062490 9639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9be666d5838941b98ea1d1da887938c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9be666d5838941b98ea1d1da887938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39413 } } }
I20260620 09:17:19.062597 9639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [sys.catalog]: This master's current role is: LEADER
I20260620 09:17:19.062628 9640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9be666d5838941b98ea1d1da887938c6. Latest consensus state: current_term: 1 leader_uuid: "9be666d5838941b98ea1d1da887938c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9be666d5838941b98ea1d1da887938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39413 } } }
I20260620 09:17:19.062677 9640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6 [sys.catalog]: This master's current role is: LEADER
I20260620 09:17:19.062837 9647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:17:19.063287 9647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:17:19.063673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 9568
I20260620 09:17:19.063757 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:17:19.063934 392 external_mini_cluster.cc:1428] Setting key 40e4ca28c55fda188dd849a266972d8a
I20260620 09:17:19.064786 9647 catalog_manager.cc:1347] Generated new cluster ID: 32b26c1fcefa40ee88cf1c841fe352be
I20260620 09:17:19.064839 9647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:17:19.068608 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.064360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.0.98.62:39413)
0620 09:17:19.064846 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:19.064855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:19.064885 (+ 30us) server_negotiation.cc:407] Connection header received
0620 09:17:19.064984 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:19.064989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:19.065045 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:19.065156 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:19.065364 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.066214 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.066932 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.067329 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.067438 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:19.068042 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:19.068045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:19.068046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:17:19.068202 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:19.068233 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:17:19.068243 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:19.068295 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:19.068375 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 09:17:19.068454 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":76,"threads_started":1}
I20260620 09:17:19.083879 9647 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:17:19.084504 9647 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:17:19.089597 9647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9be666d5838941b98ea1d1da887938c6: Generated new TSK 0
I20260620 09:17:19.089807 9647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:17:19.092126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39413
--builtin_ntp_servers=127.0.98.20:37171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 09:17:19.188210 9659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:19.188419 9659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:19.188441 9659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:19.189640 9659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:19.189680 9659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:19.189706 9659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:17:19.190994 9659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:19.191200 9659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:19.191474 9659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:19.196123 9665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:19.196135 9667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:19.196149 9664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.196568 9659 server_base.cc:1034] running on GCE node
I20260620 09:17:19.196753 9659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:19.196969 9659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:19.198103 9659 hybrid_clock.cc:648] HybridClock initialized: now 1781947039198094 us; error 33 us; skew 500 ppm
I20260620 09:17:19.199273 9659 webserver.cc:460] Webserver started at http://127.0.98.1:37281/ using document root <none> and password file <none>
I20260620 09:17:19.199477 9659 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:19.199524 9659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:19.199678 9659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:19.200703 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "1e7df819b8484a1792c867f0e5ea670a"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "a87bffacf023cdcd8cd0c304f461b63a"
server_key_iv: "fde3aff3ad84f692f5f806fd79f29a02"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.201033 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1e7df819b8484a1792c867f0e5ea670a"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "a87bffacf023cdcd8cd0c304f461b63a"
server_key_iv: "fde3aff3ad84f692f5f806fd79f29a02"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.202234 9659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:17:19.202890 9673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.203034 9659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:17:19.203117 9659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "1e7df819b8484a1792c867f0e5ea670a"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "a87bffacf023cdcd8cd0c304f461b63a"
server_key_iv: "fde3aff3ad84f692f5f806fd79f29a02"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.203176 9659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:19.210773 9659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:19.211042 9659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:19.211155 9659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:19.211349 9659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:19.211683 9659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:19.211724 9659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.211753 9659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:19.211776 9659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.217025 9659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37473
I20260620 09:17:19.217080 9786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37473 every 8 connection(s)
I20260620 09:17:19.217355 9659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:17:19.219100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 9659
I20260620 09:17:19.219172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:17:19.219347 392 external_mini_cluster.cc:1428] Setting key 8251d586da09e7e7a6fae92ede4b9c10
I20260620 09:17:19.220207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39413
--builtin_ntp_servers=127.0.98.20:37171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 09:17:19.222560 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.217988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51065 (local address 127.0.98.62:39413)
0620 09:17:19.218041 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:19.218043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:19.218270 (+ 227us) server_negotiation.cc:407] Connection header received
0620 09:17:19.218729 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:19.218730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:19.218741 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:19.218785 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:19.219450 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.220134 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.221234 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.221435 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.221487 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:19.222336 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:19.222338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:19.222338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:17:19.222342 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:19.222358 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:17:19.222362 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:19.222404 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:19.222480 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 09:17:19.222505 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:17:19.222798 9789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.217982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39413 (local address 127.0.98.1:51065)
0620 09:17:19.218162 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:19.218194 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:19.218658 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:19.218847 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:19.218851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:19.218859 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:19.219357 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:19.219363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.220214 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.220215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:19.221150 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:19.221153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.221498 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.221499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:19.221634 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:19.222186 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:17:19.222187 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:19.222225 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:17:19.222254 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:17:19.222414 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:17:19.222418 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:17:19.222421 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:17:19.222482 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 09:17:19.222598 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260620 09:17:19.223258 9787 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39413
I20260620 09:17:19.223352 9787 heartbeater.cc:464] Registering TS with master...
I20260620 09:17:19.223553 9787 heartbeater.cc:511] Master 127.0.98.62:39413 requested a full tablet report, sending...
I20260620 09:17:19.224038 9599 ts_manager.cc:194] Registered new tserver with Master: 1e7df819b8484a1792c867f0e5ea670a (127.0.98.1:37473)
I20260620 09:17:19.224628 9599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51065
W20260620 09:17:19.297801 9790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:19.297994 9790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:19.298012 9790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:19.299113 9790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:19.299150 9790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:19.299175 9790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:17:19.300453 9790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:19.300663 9790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:19.300923 9790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:19.305449 9795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.305509 9790 server_base.cc:1034] running on GCE node
W20260620 09:17:19.305480 9796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:19.305485 9798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.305846 9790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:19.306033 9790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:19.307152 9790 hybrid_clock.cc:648] HybridClock initialized: now 1781947039307147 us; error 29 us; skew 500 ppm
I20260620 09:17:19.308254 9790 webserver.cc:460] Webserver started at http://127.0.98.2:46805/ using document root <none> and password file <none>
I20260620 09:17:19.308450 9790 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:19.308491 9790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:19.308600 9790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:19.309638 9790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "b0eab965b7c94da78ee3f9fb5c7f2d3a"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "a02610d4f9e142f9309fe2263804b448"
server_key_iv: "542ce6891a67671a6c0f19464b07d262"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.309942 9790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b0eab965b7c94da78ee3f9fb5c7f2d3a"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "a02610d4f9e142f9309fe2263804b448"
server_key_iv: "542ce6891a67671a6c0f19464b07d262"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.311048 9790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:17:19.311724 9804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.311877 9790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.311951 9790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "b0eab965b7c94da78ee3f9fb5c7f2d3a"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "a02610d4f9e142f9309fe2263804b448"
server_key_iv: "542ce6891a67671a6c0f19464b07d262"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.312008 9790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:19.345129 9790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:19.345438 9790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:19.345563 9790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:19.345782 9790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:19.346097 9790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:19.346129 9790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.346158 9790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:19.346179 9790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.351187 9790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41457
I20260620 09:17:19.351251 9917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41457 every 8 connection(s)
I20260620 09:17:19.351538 9790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:17:19.355229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 9790
I20260620 09:17:19.355299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:17:19.355470 392 external_mini_cluster.cc:1428] Setting key 8a0c3afed3cb68d31ab5c80c122e9e62
I20260620 09:17:19.356413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39413
--builtin_ntp_servers=127.0.98.20:37171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 09:17:19.356551 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.352223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52673 (local address 127.0.98.62:39413)
0620 09:17:19.352278 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:19.352280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:19.352501 (+ 221us) server_negotiation.cc:407] Connection header received
0620 09:17:19.353002 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:19.353004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:19.353016 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:19.353066 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:19.353721 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.354214 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.355150 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.355350 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.355415 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:19.356310 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:19.356310 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:19.356311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:17:19.356314 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:19.356329 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:17:19.356333 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:19.356370 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:19.356446 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 09:17:19.356467 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:17:19.356760 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.352216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39413 (local address 127.0.98.2:52673)
0620 09:17:19.352390 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:19.352422 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:19.352906 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:19.353138 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:19.353141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:19.353149 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:19.353637 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:19.353641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.354291 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.354292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:19.355056 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:19.355058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.355422 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.355423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:19.355559 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:19.356165 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:17:19.356166 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:19.356204 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:17:19.356232 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:17:19.356375 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:17:19.356379 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:17:19.356382 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:17:19.356448 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:17:19.356557 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260620 09:17:19.357239 9918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39413
I20260620 09:17:19.357329 9918 heartbeater.cc:464] Registering TS with master...
I20260620 09:17:19.357515 9918 heartbeater.cc:511] Master 127.0.98.62:39413 requested a full tablet report, sending...
I20260620 09:17:19.357896 9599 ts_manager.cc:194] Registered new tserver with Master: b0eab965b7c94da78ee3f9fb5c7f2d3a (127.0.98.2:41457)
I20260620 09:17:19.358382 9599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52673
W20260620 09:17:19.434648 9921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:19.434834 9921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:19.434852 9921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:19.436005 9921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:19.436045 9921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:19.436067 9921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:17:19.437361 9921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:19.437566 9921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:19.437829 9921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:19.442440 9926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:19.442449 9927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.442474 9921 server_base.cc:1034] running on GCE node
W20260620 09:17:19.442497 9929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.442889 9921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:19.443128 9921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:19.444293 9921 hybrid_clock.cc:648] HybridClock initialized: now 1781947039444274 us; error 34 us; skew 500 ppm
I20260620 09:17:19.445554 9921 webserver.cc:460] Webserver started at http://127.0.98.3:46229/ using document root <none> and password file <none>
I20260620 09:17:19.445796 9921 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:19.445847 9921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:19.445967 9921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:19.446993 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "a176780a07504e98893873bb084d6170"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "41e1cc8e20e017e3e47ee177c01302d2"
server_key_iv: "ff8cfbac6b8cd3b8f1643c6e347657da"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.447312 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a176780a07504e98893873bb084d6170"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "41e1cc8e20e017e3e47ee177c01302d2"
server_key_iv: "ff8cfbac6b8cd3b8f1643c6e347657da"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.448554 9921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:17:19.449211 9935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.449353 9921 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:17:19.449427 9921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "a176780a07504e98893873bb084d6170"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "41e1cc8e20e017e3e47ee177c01302d2"
server_key_iv: "ff8cfbac6b8cd3b8f1643c6e347657da"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.449486 9921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:19.464160 9921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:19.464453 9921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:19.464573 9921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:19.464788 9921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:19.465099 9921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:19.465133 9921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.465162 9921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:19.465186 9921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.470487 9921 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37263
I20260620 09:17:19.470533 10048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37263 every 8 connection(s)
I20260620 09:17:19.470832 9921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:17:19.471436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 9921
I20260620 09:17:19.471525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:17:19.471742 392 external_mini_cluster.cc:1428] Setting key 6bcbe6a40aca3dc9ce54cb5dea3928f8
I20260620 09:17:19.476044 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.471512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39727 (local address 127.0.98.62:39413)
0620 09:17:19.471554 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:19.471555 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:19.471923 (+ 368us) server_negotiation.cc:407] Connection header received
0620 09:17:19.472518 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:19.472519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:19.472530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:19.472578 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:19.473236 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.473724 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.474669 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.474853 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.474913 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:19.475774 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:19.475776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:19.475776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:17:19.475781 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:19.475797 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:17:19.475801 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:19.475844 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:19.475927 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 09:17:19.475962 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:17:19.476250 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.471530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39413 (local address 127.0.98.3:39727)
0620 09:17:19.471789 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:19.471836 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:19.472424 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:19.472661 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:19.472665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:19.472674 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:19.473155 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:19.473158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.473800 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.473801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:19.474591 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:19.474593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:19.474924 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:19.474925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:19.475062 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:19.475605 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:17:19.475606 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:19.475665 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:17:19.475694 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:17:19.475850 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:17:19.475855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:17:19.475859 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:17:19.475928 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 09:17:19.476033 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":109,"threads_started":1}
I20260620 09:17:19.476670 10049 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39413
I20260620 09:17:19.476766 10049 heartbeater.cc:464] Registering TS with master...
I20260620 09:17:19.476938 10049 heartbeater.cc:511] Master 127.0.98.62:39413 requested a full tablet report, sending...
I20260620 09:17:19.477259 9599 ts_manager.cc:194] Registered new tserver with Master: a176780a07504e98893873bb084d6170 (127.0.98.3:37263)
I20260620 09:17:19.477605 9599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39727
I20260620 09:17:19.483265 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 09:17:19.485056 9657 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:19.484539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.0.98.62:39413)
0620 09:17:19.484584 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:19.484586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:19.484761 (+ 175us) server_negotiation.cc:407] Connection header received
0620 09:17:19.484816 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:19.484817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:19.484829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:19.484867 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:19.484989 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48872: BlockingRecv error: recv got EOF from 127.0.0.1:48872 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:17:19.485138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 9659
I20260620 09:17:19.489385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 9790
I20260620 09:17:19.492697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 9921
I20260620 09:17:19.496150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 9568
2026-06-20T09:17:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (580 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T09:17:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:17:19Z Disabled control of system clock
I20260620 09:17:19.519554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43495
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:17:19.596036 10060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:19.596238 10060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:19.596257 10060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:19.597309 10060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:17:19.597340 10060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:19.597352 10060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:19.597362 10060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:17:19.597373 10060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:17:19.598592 10060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43495
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:19.598785 10060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:19.599015 10060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:17:19.603469 10060 server_base.cc:1034] running on GCE node
W20260620 09:17:19.603406 10065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:19.603402 10068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:19.603415 10066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.603916 10060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:19.604183 10060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:19.605345 10060 hybrid_clock.cc:648] HybridClock initialized: now 1781947039605321 us; error 33 us; skew 500 ppm
I20260620 09:17:19.606463 10060 webserver.cc:460] Webserver started at http://127.0.98.62:41391/ using document root <none> and password file <none>
I20260620 09:17:19.606662 10060 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:19.606709 10060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:19.606834 10060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:19.607841 10060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "dbe8201c51f545a2b4889010eeadcf15"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "29f942465e20d70c91b4d5f8e3756704"
server_key_iv: "6f54fe1c4fb9b20901166dbdb2bf9713"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.608144 10060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "dbe8201c51f545a2b4889010eeadcf15"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "29f942465e20d70c91b4d5f8e3756704"
server_key_iv: "6f54fe1c4fb9b20901166dbdb2bf9713"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.609265 10060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:17:19.609865 10074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.610004 10060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.610080 10060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "dbe8201c51f545a2b4889010eeadcf15"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "29f942465e20d70c91b4d5f8e3756704"
server_key_iv: "6f54fe1c4fb9b20901166dbdb2bf9713"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.610136 10060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:19.613636 10060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:19.613889 10060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:19.614004 10060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:19.617427 10060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43495
I20260620 09:17:19.617456 10126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43495 every 8 connection(s)
I20260620 09:17:19.617758 10060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:17:19.618341 10127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:17:19.620888 10127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15: Bootstrap starting.
I20260620 09:17:19.621394 10127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15: Neither blocks nor log segments found. Creating new log.
I20260620 09:17:19.621639 10127 log.cc:824] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15: Log is configured to *not* fsync() on all Append() calls
I20260620 09:17:19.622228 10127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15: No bootstrap required, opened a new log
I20260620 09:17:19.623016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 10060
I20260620 09:17:19.623098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:17:19.623289 392 external_mini_cluster.cc:1428] Setting key 03d3686c740afd26bb9effd2c95f4d2e
I20260620 09:17:19.624037 10127 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe8201c51f545a2b4889010eeadcf15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43495 } }
I20260620 09:17:19.624171 10127 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:17:19.624202 10127 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe8201c51f545a2b4889010eeadcf15, State: Initialized, Role: FOLLOWER
I20260620 09:17:19.624327 10127 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe8201c51f545a2b4889010eeadcf15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43495 } }
I20260620 09:17:19.624397 10127 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:17:19.624436 10127 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:17:19.624480 10127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:17:19.625020 10127 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe8201c51f545a2b4889010eeadcf15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43495 } }
I20260620 09:17:19.625120 10127 leader_election.cc:302] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbe8201c51f545a2b4889010eeadcf15; no voters:
I20260620 09:17:19.625267 10127 leader_election.cc:288] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:17:19.625291 10132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:17:19.625396 10132 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [term 1 LEADER]: Becoming Leader. State: Replica: dbe8201c51f545a2b4889010eeadcf15, State: Running, Role: LEADER
I20260620 09:17:19.625516 10127 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:17:19.625524 10132 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe8201c51f545a2b4889010eeadcf15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43495 } }
I20260620 09:17:19.626026 10132 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbe8201c51f545a2b4889010eeadcf15. Latest consensus state: current_term: 1 leader_uuid: "dbe8201c51f545a2b4889010eeadcf15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe8201c51f545a2b4889010eeadcf15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43495 } } }
I20260620 09:17:19.626034 10133 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbe8201c51f545a2b4889010eeadcf15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe8201c51f545a2b4889010eeadcf15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43495 } } }
I20260620 09:17:19.626111 10132 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [sys.catalog]: This master's current role is: LEADER
I20260620 09:17:19.626116 10133 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15 [sys.catalog]: This master's current role is: LEADER
I20260620 09:17:19.626196 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.623822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.0.98.62:43495)
0620 09:17:19.623996 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:19.624002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:19.624023 (+ 21us) server_negotiation.cc:407] Connection header received
0620 09:17:19.624116 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:19.624121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:19.624165 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:19.624290 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:19.625322 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:19.625326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:19.625326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:17:19.625330 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:19.625376 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:17:19.625391 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:19.625452 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:19.625522 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 09:17:19.625576 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260620 09:17:19.626289 10138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:17:19.627151 10138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:17:19.628443 10138 catalog_manager.cc:1347] Generated new cluster ID: 8e11555cc9674acd909eeb7d5b18e6eb
I20260620 09:17:19.628490 10138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:17:19.633790 10138 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:17:19.633940 10138 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:17:19.641384 10138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbe8201c51f545a2b4889010eeadcf15: Generated new TSK 0
I20260620 09:17:19.641597 10138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:17:19.646159 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43495
--builtin_ntp_servers=127.0.98.20:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:17:19.742460 10151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:19.742648 10151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:19.742667 10151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:19.743788 10151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:19.743824 10151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:19.743847 10151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:17:19.745111 10151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:19.745334 10151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:19.745695 10151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:19.751241 10157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:19.751262 10156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.751480 10151 server_base.cc:1034] running on GCE node
W20260620 09:17:19.751242 10159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.751771 10151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:19.752040 10151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:19.753209 10151 hybrid_clock.cc:648] HybridClock initialized: now 1781947039753198 us; error 28 us; skew 500 ppm
I20260620 09:17:19.756886 10151 webserver.cc:460] Webserver started at http://127.0.98.1:43973/ using document root <none> and password file <none>
I20260620 09:17:19.757138 10151 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:19.757195 10151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:19.757308 10151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:19.758592 10151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "2332904a9a5b43d9aad0d8bf89636279"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "486fb9856526da7bec59a9d12a605c36"
server_key_iv: "66e11c6e877d950595eb5800a48376e9"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.758968 10151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2332904a9a5b43d9aad0d8bf89636279"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "486fb9856526da7bec59a9d12a605c36"
server_key_iv: "66e11c6e877d950595eb5800a48376e9"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.760480 10151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:17:19.761283 10165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.761442 10151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:17:19.761529 10151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "2332904a9a5b43d9aad0d8bf89636279"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "486fb9856526da7bec59a9d12a605c36"
server_key_iv: "66e11c6e877d950595eb5800a48376e9"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.761597 10151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:19.766377 10151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:19.766639 10151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:19.766750 10151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:19.766939 10151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:19.767226 10151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:19.767266 10151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.767298 10151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:19.767321 10151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.773757 10151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36103
I20260620 09:17:19.773811 10278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36103 every 8 connection(s)
I20260620 09:17:19.774145 10151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:17:19.776862 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.774826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44063 (local address 127.0.98.62:43495)
0620 09:17:19.774897 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:19.774899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:19.775126 (+ 227us) server_negotiation.cc:407] Connection header received
0620 09:17:19.775593 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:19.775594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:19.775600 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:19.775701 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:19.776682 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:19.776683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:19.776683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:17:19.776687 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:19.776703 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:17:19.776707 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:19.776749 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:19.776798 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 09:17:19.776810 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:17:19.777031 10281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.774809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43495 (local address 127.0.98.1:44063)
0620 09:17:19.774991 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:19.775037 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:19.775504 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:19.775754 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:19.775757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:19.775765 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:19.776557 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:17:19.776558 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:19.776584 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:17:19.776609 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:17:19.776753 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:17:19.776755 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:17:19.776758 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:17:19.776803 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:17:19.776859 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260620 09:17:19.777446 10279 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43495
I20260620 09:17:19.777529 10279 heartbeater.cc:464] Registering TS with master...
I20260620 09:17:19.777663 10279 heartbeater.cc:511] Master 127.0.98.62:43495 requested a full tablet report, sending...
I20260620 09:17:19.778095 10091 ts_manager.cc:194] Registered new tserver with Master: 2332904a9a5b43d9aad0d8bf89636279 (127.0.98.1:36103)
I20260620 09:17:19.784370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 10151
I20260620 09:17:19.784442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:17:19.784585 392 external_mini_cluster.cc:1428] Setting key 624593af4f0cf051c67383fb004a761c
I20260620 09:17:19.785414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43495
--builtin_ntp_servers=127.0.98.20:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:17:19.863126 10282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:19.863336 10282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:19.863358 10282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:19.864562 10282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:19.864616 10282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:19.864660 10282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:17:19.866055 10282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:19.866247 10282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:19.866506 10282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:19.871238 10288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:19.871235 10290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.871338 10282 server_base.cc:1034] running on GCE node
W20260620 09:17:19.871228 10287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.871750 10282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:19.872005 10282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:19.873169 10282 hybrid_clock.cc:648] HybridClock initialized: now 1781947039873147 us; error 35 us; skew 500 ppm
I20260620 09:17:19.874506 10282 webserver.cc:460] Webserver started at http://127.0.98.2:41063/ using document root <none> and password file <none>
I20260620 09:17:19.874739 10282 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:19.874794 10282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:19.874923 10282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:19.876044 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "93b044101a1b419782bfe897e6f227bf"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "3815f3a322aea2bfb7019ea5fccdc745"
server_key_iv: "2ba4be986021c4fcdd9048f5a2f98a51"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.876358 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "93b044101a1b419782bfe897e6f227bf"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "3815f3a322aea2bfb7019ea5fccdc745"
server_key_iv: "2ba4be986021c4fcdd9048f5a2f98a51"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.877540 10282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:17:19.878187 10296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.878324 10282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:17:19.878396 10282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "93b044101a1b419782bfe897e6f227bf"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "3815f3a322aea2bfb7019ea5fccdc745"
server_key_iv: "2ba4be986021c4fcdd9048f5a2f98a51"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.878458 10282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:19.881657 10282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:19.881881 10282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:19.881990 10282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:19.882167 10282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:19.882423 10282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:19.882453 10282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.882479 10282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:19.882503 10282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.887732 10282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42501
I20260620 09:17:19.887790 10409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42501 every 8 connection(s)
I20260620 09:17:19.888055 10282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:17:19.889200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 10282
I20260620 09:17:19.889282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:17:19.889442 392 external_mini_cluster.cc:1428] Setting key 123fd989088488959d2bb48fd6e7ed6f
I20260620 09:17:19.890228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43495
--builtin_ntp_servers=127.0.98.20:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 09:17:19.890771 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.888665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34717 (local address 127.0.98.62:43495)
0620 09:17:19.888722 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:19.888724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:19.888925 (+ 201us) server_negotiation.cc:407] Connection header received
0620 09:17:19.889390 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:19.889391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:19.889398 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:19.889453 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:19.890540 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:19.890541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:19.890542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:17:19.890546 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:19.890562 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:17:19.890567 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:19.890615 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:19.890691 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 09:17:19.890705 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:17:19.890949 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.888653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43495 (local address 127.0.98.2:34717)
0620 09:17:19.888815 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:19.888852 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:19.889269 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:19.889523 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:19.889527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:19.889537 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:19.890404 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:17:19.890405 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:19.890439 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:17:19.890468 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:17:19.890623 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:17:19.890628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:17:19.890631 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:17:19.890691 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:17:19.890749 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260620 09:17:19.891430 10410 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43495
I20260620 09:17:19.891510 10410 heartbeater.cc:464] Registering TS with master...
I20260620 09:17:19.891666 10410 heartbeater.cc:511] Master 127.0.98.62:43495 requested a full tablet report, sending...
I20260620 09:17:19.891997 10091 ts_manager.cc:194] Registered new tserver with Master: 93b044101a1b419782bfe897e6f227bf (127.0.98.2:42501)
W20260620 09:17:19.969583 10413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:19.969781 10413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:19.969810 10413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:19.970961 10413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:19.970999 10413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:19.971032 10413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:17:19.972378 10413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:19.972606 10413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:19.972860 10413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:19.977419 10419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:19.977464 10418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.977594 10413 server_base.cc:1034] running on GCE node
W20260620 09:17:19.977476 10421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:19.977838 10413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:19.978029 10413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:19.979156 10413 hybrid_clock.cc:648] HybridClock initialized: now 1781947039979145 us; error 26 us; skew 500 ppm
I20260620 09:17:19.980270 10413 webserver.cc:460] Webserver started at http://127.0.98.3:34051/ using document root <none> and password file <none>
I20260620 09:17:19.980472 10413 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:19.980513 10413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:19.980626 10413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:19.981654 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "66b2ae7af0cb4705b73dc90e621fc8c6"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "3ced2c10697377f32e987d916b382f9a"
server_key_iv: "f6868c35be9b453a38e193bc11d03e50"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.981983 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "66b2ae7af0cb4705b73dc90e621fc8c6"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "3ced2c10697377f32e987d916b382f9a"
server_key_iv: "f6868c35be9b453a38e193bc11d03e50"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.983111 10413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:17:19.983758 10427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.983906 10413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:17:19.983978 10413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "66b2ae7af0cb4705b73dc90e621fc8c6"
format_stamp: "Formatted at 2026-06-20 09:17:19 on dist-test-slave-rq8c"
server_key: "3ced2c10697377f32e987d916b382f9a"
server_key_iv: "f6868c35be9b453a38e193bc11d03e50"
server_key_version: "encryptionkey@0"
I20260620 09:17:19.984043 10413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:19.987239 10413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:19.987469 10413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:19.987577 10413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:19.987771 10413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:19.988015 10413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:19.988044 10413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.988062 10413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:19.988078 10413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:19.993351 10413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34835
I20260620 09:17:19.993391 10540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34835 every 8 connection(s)
I20260620 09:17:19.993683 10413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:17:19.994597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 10413
I20260620 09:17:19.994681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:17:19.994855 392 external_mini_cluster.cc:1428] Setting key 16c7063a43595dd904b257bb411205b0
I20260620 09:17:19.996595 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.994351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42741 (local address 127.0.98.62:43495)
0620 09:17:19.994402 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:19.994404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:19.994592 (+ 188us) server_negotiation.cc:407] Connection header received
0620 09:17:19.995113 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:19.995114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:19.995120 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:19.995178 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:19.996419 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:19.996421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:19.996421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:17:19.996424 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:19.996438 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:17:19.996442 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:19.996486 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:19.996530 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 09:17:19.996542 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:17:19.996766 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:19.994334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43495 (local address 127.0.98.3:42741)
0620 09:17:19.994490 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:19.994521 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:19.995036 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:19.995312 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:19.995316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:19.995326 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:19.996284 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:17:19.996285 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:19.996314 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:17:19.996342 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:17:19.996485 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:17:19.996487 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:17:19.996489 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:17:19.996536 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:17:19.996583 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260620 09:17:19.997167 10541 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43495
I20260620 09:17:19.997239 10541 heartbeater.cc:464] Registering TS with master...
I20260620 09:17:19.997370 10541 heartbeater.cc:511] Master 127.0.98.62:43495 requested a full tablet report, sending...
I20260620 09:17:19.997653 10091 ts_manager.cc:194] Registered new tserver with Master: 66b2ae7af0cb4705b73dc90e621fc8c6 (127.0.98.3:34835)
I20260620 09:17:20.006731 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 09:17:20.008383 10131 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:17:20.007968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55188 (local address 127.0.98.62:43495)
0620 09:17:20.008020 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:20.008022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:20.008112 (+ 90us) server_negotiation.cc:407] Connection header received
0620 09:17:20.008163 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:20.008163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:20.008167 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:20.008213 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:20.008324 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55188: BlockingRecv error: recv got EOF from 127.0.0.1:55188 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:17:20.008455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 10151
I20260620 09:17:20.012123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 10282
I20260620 09:17:20.015595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 10413
I20260620 09:17:20.019101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 10060
2026-06-20T09:17:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (523 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:17:20 dist-test-slave-rq8c krb5kdc[10551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:17:20 dist-test-slave-rq8c krb5kdc[10551](info): set up 2 sockets
Jun 20 09:17:20 dist-test-slave-rq8c krb5kdc[10551](info): commencing operation
krb5kdc: starting...
W20260620 09:17:20.968263 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:17:21 dist-test-slave-rq8c krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:17:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:17:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:17:21.074100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:17:21.157527 10564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:21.157712 10564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:21.157734 10564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:21.158818 10564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:17:21.158850 10564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:21.158864 10564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:21.158874 10564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:17:21.158885 10564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:17:21.160320 10564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:21.160503 10564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:21.160663 10564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:21.165007 10569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:21.165014 10570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:21.165189 10564 server_base.cc:1034] running on GCE node
W20260620 09:17:21.165073 10572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:21.165438 10564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:21.165697 10564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:21.166872 10564 hybrid_clock.cc:648] HybridClock initialized: now 1781947041166858 us; error 28 us; skew 500 ppm
Jun 20 09:17:21 dist-test-slave-rq8c krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:17:21.169312 10564 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:17:21.169727 10564 webserver.cc:460] Webserver started at http://127.0.98.62:39115/ using document root <none> and password file <none>
I20260620 09:17:21.169924 10564 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:21.169972 10564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:21.170120 10564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:21.171113 10564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "b21a2d76407a45afbc7074ad72fa2fa8"
format_stamp: "Formatted at 2026-06-20 09:17:21 on dist-test-slave-rq8c"
server_key: "378d9825d1adefa3e742f004ca46a79c"
server_key_iv: "6afd5d1ece115aee95b89a2be7ecad6a"
server_key_version: "encryptionkey@0"
I20260620 09:17:21.171458 10564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "b21a2d76407a45afbc7074ad72fa2fa8"
format_stamp: "Formatted at 2026-06-20 09:17:21 on dist-test-slave-rq8c"
server_key: "378d9825d1adefa3e742f004ca46a79c"
server_key_iv: "6afd5d1ece115aee95b89a2be7ecad6a"
server_key_version: "encryptionkey@0"
I20260620 09:17:21.172767 10564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:17:21.173449 10579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:21.173591 10564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:17:21.173676 10564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "b21a2d76407a45afbc7074ad72fa2fa8"
format_stamp: "Formatted at 2026-06-20 09:17:21 on dist-test-slave-rq8c"
server_key: "378d9825d1adefa3e742f004ca46a79c"
server_key_iv: "6afd5d1ece115aee95b89a2be7ecad6a"
server_key_version: "encryptionkey@0"
I20260620 09:17:21.173745 10564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:21.192925 10564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:21.194698 10564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:21.194837 10564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:21.198696 10564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41775
I20260620 09:17:21.198742 10631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41775 every 8 connection(s)
I20260620 09:17:21.199061 10564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:17:21.199990 10632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:17:21.202121 10632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8: Bootstrap starting.
I20260620 09:17:21.202673 10632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8: Neither blocks nor log segments found. Creating new log.
I20260620 09:17:21.202919 10632 log.cc:824] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8: Log is configured to *not* fsync() on all Append() calls
I20260620 09:17:21.203550 10632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8: No bootstrap required, opened a new log
I20260620 09:17:21.205078 10632 raft_consensus.cc:348] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21a2d76407a45afbc7074ad72fa2fa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41775 } }
I20260620 09:17:21.205222 10632 raft_consensus.cc:374] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:17:21.205255 10632 raft_consensus.cc:729] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21a2d76407a45afbc7074ad72fa2fa8, State: Initialized, Role: FOLLOWER
I20260620 09:17:21.205395 10632 consensus_queue.cc:260] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21a2d76407a45afbc7074ad72fa2fa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41775 } }
I20260620 09:17:21.205454 10632 raft_consensus.cc:388] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:17:21.205476 10632 raft_consensus.cc:482] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:17:21.205519 10632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:17:21.206250 10632 raft_consensus.cc:504] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21a2d76407a45afbc7074ad72fa2fa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41775 } }
I20260620 09:17:21.206338 10632 leader_election.cc:302] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b21a2d76407a45afbc7074ad72fa2fa8; no voters:
I20260620 09:17:21.206483 10632 leader_election.cc:288] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:17:21.206518 10635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:17:21.206661 10635 raft_consensus.cc:686] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [term 1 LEADER]: Becoming Leader. State: Replica: b21a2d76407a45afbc7074ad72fa2fa8, State: Running, Role: LEADER
I20260620 09:17:21.206708 10632 sys_catalog.cc:564] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:17:21.206755 10635 consensus_queue.cc:237] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21a2d76407a45afbc7074ad72fa2fa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41775 } }
I20260620 09:17:21.207068 10636 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b21a2d76407a45afbc7074ad72fa2fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21a2d76407a45afbc7074ad72fa2fa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41775 } } }
I20260620 09:17:21.207083 10637 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b21a2d76407a45afbc7074ad72fa2fa8. Latest consensus state: current_term: 1 leader_uuid: "b21a2d76407a45afbc7074ad72fa2fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21a2d76407a45afbc7074ad72fa2fa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41775 } } }
I20260620 09:17:21.207190 10636 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:17:21.207213 10637 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8 [sys.catalog]: This master's current role is: LEADER
I20260620 09:17:21.207405 10643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:17:21.207924 10643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:17:21.208148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 10564
I20260620 09:17:21.208245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:17:21.208425 392 external_mini_cluster.cc:1428] Setting key 1da7b20ffb87c589cd68da2ee06c8db6
I20260620 09:17:21.209450 10643 catalog_manager.cc:1347] Generated new cluster ID: a05dcafcbb064580ab879b67ff2f0a21
I20260620 09:17:21.209494 10643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:17:21 dist-test-slave-rq8c krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:17:21.217031 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.208992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.0.98.62:41775)
0620 09:17:21.209141 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:21.209148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:21.209172 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:17:21.209270 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:21.209275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:21.209316 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:21.209471 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:21.210049 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.210928 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.211678 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.212107 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.212208 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.214220 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:21.214223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:21.214224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:17:21.214235 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:21.215734 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:21.216026 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:21.216028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:21.216029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:21.216124 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:21.216296 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:21.216296 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:21.216297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:21.216514 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:21.216651 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:21.216746 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 09:17:21.216874 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260620 09:17:21.218991 10643 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:17:21.219457 10643 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:17:21.229029 10643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8: Generated new TSK 0
I20260620 09:17:21.229187 10643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:17:21.253943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41775
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:17:21.331355 10658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:21.331542 10658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:21.331564 10658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:21.332669 10658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:21.332707 10658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:21.332731 10658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:17:21.334084 10658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:21.334268 10658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:21.334434 10658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:21.338970 10666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:21.339145 10658 server_base.cc:1034] running on GCE node
W20260620 09:17:21.338979 10664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:21.339020 10663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:21.339416 10658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:21.339651 10658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:21.340787 10658 hybrid_clock.cc:648] HybridClock initialized: now 1781947041340775 us; error 30 us; skew 500 ppm
Jun 20 09:17:21 dist-test-slave-rq8c krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:17:21.343060 10658 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:17:21.343431 10658 webserver.cc:460] Webserver started at http://127.0.98.1:45627/ using document root <none> and password file <none>
I20260620 09:17:21.343603 10658 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:21.343681 10658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:21.343780 10658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:21.344715 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "2b78ee6ff99f4a7298a3c90e60360844"
format_stamp: "Formatted at 2026-06-20 09:17:21 on dist-test-slave-rq8c"
server_key: "95db46f39a49b40d78ac581d5d55e865"
server_key_iv: "17e9614d4afd83279b17d70e8aaa5aa1"
server_key_version: "encryptionkey@0"
I20260620 09:17:21.344990 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2b78ee6ff99f4a7298a3c90e60360844"
format_stamp: "Formatted at 2026-06-20 09:17:21 on dist-test-slave-rq8c"
server_key: "95db46f39a49b40d78ac581d5d55e865"
server_key_iv: "17e9614d4afd83279b17d70e8aaa5aa1"
server_key_version: "encryptionkey@0"
I20260620 09:17:21.346088 10658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:17:21.346688 10673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:21.346832 10658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:17:21.346908 10658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "2b78ee6ff99f4a7298a3c90e60360844"
format_stamp: "Formatted at 2026-06-20 09:17:21 on dist-test-slave-rq8c"
server_key: "95db46f39a49b40d78ac581d5d55e865"
server_key_iv: "17e9614d4afd83279b17d70e8aaa5aa1"
server_key_version: "encryptionkey@0"
I20260620 09:17:21.346961 10658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:21.363415 10658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:21.365130 10658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:21.365235 10658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:21.365407 10658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:21.365646 10658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:21.365674 10658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:21.365694 10658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:21.365710 10658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:21.370970 10658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36533
I20260620 09:17:21.371037 10786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36533 every 8 connection(s)
I20260620 09:17:21.371311 10658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:17:21 dist-test-slave-rq8c krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:17:21.378118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 10658
I20260620 09:17:21.378211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:17:21.378398 392 external_mini_cluster.cc:1428] Setting key bff16cd9b0639e2752867237777fc24f
I20260620 09:17:21.380029 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.371993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52737 (local address 127.0.98.62:41775)
0620 09:17:21.372053 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:21.372055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:21.372244 (+ 189us) server_negotiation.cc:407] Connection header received
0620 09:17:21.372723 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:21.372724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:21.372739 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:21.372789 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:21.373690 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.374189 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.375122 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.375329 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.375382 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.377409 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:21.377411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:21.377411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:17:21.377418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:21.378732 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:21.379138 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:21.379139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:21.379139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:21.379204 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:21.379410 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:21.379411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:21.379411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:21.379535 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:21.379599 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:21.379863 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 09:17:21.379944 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:17:21.380122 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.371966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41775 (local address 127.0.98.1:52737)
0620 09:17:21.372127 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:21.372161 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:21.372635 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:21.372857 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:21.372861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:21.373407 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:21.373593 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:21.373598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.374265 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.374266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:21.375044 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:21.375046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.375393 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.375393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:21.375516 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.375975 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:21.375983 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:21.377325 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:17:21.378801 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:17:21.378807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:17:21.378809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:17:21.379083 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:17:21.379261 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:17:21.379262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:17:21.379262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:17:21.379363 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:17:21.379589 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:17:21.379591 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:17:21.379752 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 09:17:21.379835 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 09:17:21.379951 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260620 09:17:21.380595 10787 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41775
I20260620 09:17:21.380686 10787 heartbeater.cc:464] Registering TS with master...
I20260620 09:17:21.380848 10787 heartbeater.cc:511] Master 127.0.98.62:41775 requested a full tablet report, sending...
I20260620 09:17:21.381323 10596 ts_manager.cc:194] Registered new tserver with Master: 2b78ee6ff99f4a7298a3c90e60360844 (127.0.98.1:36533)
I20260620 09:17:21.381902 10596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52737
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:17:21.399878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41775
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:17:21.477769 10792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:21.477955 10792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:21.477982 10792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:21.479068 10792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:21.479105 10792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:21.479137 10792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:17:21.480576 10792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:21.480796 10792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:21.480986 10792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:21.485558 10797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:21.485548 10798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:21.485548 10800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:21.485734 10792 server_base.cc:1034] running on GCE node
I20260620 09:17:21.485936 10792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:21.486183 10792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:21.487353 10792 hybrid_clock.cc:648] HybridClock initialized: now 1781947041487335 us; error 30 us; skew 500 ppm
Jun 20 09:17:21 dist-test-slave-rq8c krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:17:21.490419 10792 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:17:21.490940 10792 webserver.cc:460] Webserver started at http://127.0.98.2:37201/ using document root <none> and password file <none>
I20260620 09:17:21.491204 10792 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:21.491268 10792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:21.491401 10792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:21.492775 10792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "bc22423bbdd847eba6fa8a9141a8db27"
format_stamp: "Formatted at 2026-06-20 09:17:21 on dist-test-slave-rq8c"
server_key: "ce305102b105adee183e7010d0f572d5"
server_key_iv: "5319e87098b0c8c7bb977ab60f749bcd"
server_key_version: "encryptionkey@0"
I20260620 09:17:21.493161 10792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bc22423bbdd847eba6fa8a9141a8db27"
format_stamp: "Formatted at 2026-06-20 09:17:21 on dist-test-slave-rq8c"
server_key: "ce305102b105adee183e7010d0f572d5"
server_key_iv: "5319e87098b0c8c7bb977ab60f749bcd"
server_key_version: "encryptionkey@0"
I20260620 09:17:21.494589 10792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:17:21.495211 10807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:21.495348 10792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:17:21.495427 10792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "bc22423bbdd847eba6fa8a9141a8db27"
format_stamp: "Formatted at 2026-06-20 09:17:21 on dist-test-slave-rq8c"
server_key: "ce305102b105adee183e7010d0f572d5"
server_key_iv: "5319e87098b0c8c7bb977ab60f749bcd"
server_key_version: "encryptionkey@0"
I20260620 09:17:21.495491 10792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:21.509289 10792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:21.510905 10792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:21.511019 10792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:21.511202 10792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:21.511472 10792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:21.511503 10792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:21.511530 10792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:21.511546 10792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:21.516788 10792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33417
I20260620 09:17:21.516836 10920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33417 every 8 connection(s)
I20260620 09:17:21.517096 10792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:17:21 dist-test-slave-rq8c krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:17:21.523764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 10792
I20260620 09:17:21.523845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:17:21.524010 392 external_mini_cluster.cc:1428] Setting key e41a7b289b2f87c432145a3afadf58ff
I20260620 09:17:21.524987 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.517685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56799 (local address 127.0.98.62:41775)
0620 09:17:21.517729 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:21.517731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:21.517916 (+ 185us) server_negotiation.cc:407] Connection header received
0620 09:17:21.518372 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:21.518373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:21.518382 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:21.518420 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:21.519263 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.519781 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.520694 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.520870 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.520910 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.522676 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:21.522677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:21.522677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:17:21.522683 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:21.523742 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:21.524111 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:21.524112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:21.524113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:21.524179 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:21.524402 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:21.524403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:21.524403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:21.524522 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:21.524585 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:21.524865 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 09:17:21.524928 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:17:21.525270 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.517663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41775 (local address 127.0.98.2:56799)
0620 09:17:21.517807 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:21.517841 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:21.518290 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:21.518479 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:21.518483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:21.519005 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:21.519199 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:21.519203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.519841 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.519842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:21.520635 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:21.520637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.520917 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.520918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:21.521011 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.521403 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:21.521411 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:21.522587 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:17:21.523816 (+ 1229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:17:21.523821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:17:21.523823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:17:21.524048 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:17:21.524244 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:17:21.524244 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:17:21.524245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:17:21.524343 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:17:21.524575 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:17:21.524581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:17:21.524736 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 09:17:21.524852 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 09:17:21.524996 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260620 09:17:21.525668 10921 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41775
I20260620 09:17:21.525754 10921 heartbeater.cc:464] Registering TS with master...
I20260620 09:17:21.525934 10921 heartbeater.cc:511] Master 127.0.98.62:41775 requested a full tablet report, sending...
I20260620 09:17:21.526288 10596 ts_manager.cc:194] Registered new tserver with Master: bc22423bbdd847eba6fa8a9141a8db27 (127.0.98.2:33417)
I20260620 09:17:21.526737 10596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56799
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:17:21.543751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41775
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:17:21.620214 10926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:17:21.620391 10926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:17:21.620412 10926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:17:21.621467 10926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:17:21.621500 10926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:17:21.621523 10926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:17:21.622856 10926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:17:21.623019 10926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:17:21.623188 10926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:17:21.627646 10931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:17:21.627662 10934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:21.627887 10926 server_base.cc:1034] running on GCE node
W20260620 09:17:21.628477 10932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:17:21.628672 10926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:17:21.628859 10926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:17:21.629977 10926 hybrid_clock.cc:648] HybridClock initialized: now 1781947041629969 us; error 28 us; skew 500 ppm
Jun 20 09:17:21 dist-test-slave-rq8c krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:17:21.632270 10926 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:17:21.632697 10926 webserver.cc:460] Webserver started at http://127.0.98.3:43251/ using document root <none> and password file <none>
I20260620 09:17:21.632877 10926 fs_manager.cc:359] Metadata directory not provided
I20260620 09:17:21.632918 10926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:17:21.633037 10926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:17:21.634003 10926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "9dbb55e4483045f18cb9f245518f79df"
format_stamp: "Formatted at 2026-06-20 09:17:21 on dist-test-slave-rq8c"
server_key: "bf853ce733116e1b150cad0ce0c57800"
server_key_iv: "7e4b17e088819901872548c0c3c8cec5"
server_key_version: "encryptionkey@0"
I20260620 09:17:21.634318 10926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9dbb55e4483045f18cb9f245518f79df"
format_stamp: "Formatted at 2026-06-20 09:17:21 on dist-test-slave-rq8c"
server_key: "bf853ce733116e1b150cad0ce0c57800"
server_key_iv: "7e4b17e088819901872548c0c3c8cec5"
server_key_version: "encryptionkey@0"
I20260620 09:17:21.635469 10926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:17:21.636153 10941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:21.636294 10926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:21.636368 10926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "9dbb55e4483045f18cb9f245518f79df"
format_stamp: "Formatted at 2026-06-20 09:17:21 on dist-test-slave-rq8c"
server_key: "bf853ce733116e1b150cad0ce0c57800"
server_key_iv: "7e4b17e088819901872548c0c3c8cec5"
server_key_version: "encryptionkey@0"
I20260620 09:17:21.636433 10926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:17:21.652555 10926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:17:21.654587 10926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:17:21.654718 10926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:17:21.654908 10926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:17:21.655181 10926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:17:21.655218 10926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:21.655251 10926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:17:21.655277 10926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:17:21.660614 10926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42057
I20260620 09:17:21.660660 11054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42057 every 8 connection(s)
I20260620 09:17:21.660929 10926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:17:21 dist-test-slave-rq8c krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:17:21.667454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 10926
I20260620 09:17:21.667557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:17:21.667783 392 external_mini_cluster.cc:1428] Setting key 95af16cd193b44313f268726caef522a
I20260620 09:17:21.669245 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.661549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42415 (local address 127.0.98.62:41775)
0620 09:17:21.661590 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:21.661591 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:21.661763 (+ 172us) server_negotiation.cc:407] Connection header received
0620 09:17:21.662211 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:21.662212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:21.662220 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:21.662277 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:21.663113 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.663593 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.664550 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.664735 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.664775 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.666713 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:21.666715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:21.666715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:17:21.666721 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:21.667961 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:21.668372 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:21.668374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:21.668374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:21.668435 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:21.668668 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:21.668669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:21.668669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:21.668783 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:21.668838 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:21.669082 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 09:17:21.669159 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:17:21.669404 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.661523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41775 (local address 127.0.98.3:42415)
0620 09:17:21.661663 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:21.661702 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:21.662138 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:21.662331 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:21.662335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:21.662862 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:17:21.663045 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:21.663050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.663705 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.663706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:21.664485 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:21.664487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.664786 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.664786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:21.664898 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.665334 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:17:21.665341 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:17:21.666632 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:17:21.668036 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:17:21.668042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:17:21.668044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:17:21.668307 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:17:21.668485 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:17:21.668486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:17:21.668486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:17:21.668609 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:17:21.668835 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:17:21.668841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:17:21.668967 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 09:17:21.669053 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:17:21.669205 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260620 09:17:21.669806 11055 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41775
I20260620 09:17:21.669890 11055 heartbeater.cc:464] Registering TS with master...
I20260620 09:17:21.670048 11055 heartbeater.cc:511] Master 127.0.98.62:41775 requested a full tablet report, sending...
I20260620 09:17:21.670372 10596 ts_manager.cc:194] Registered new tserver with Master: 9dbb55e4483045f18cb9f245518f79df (127.0.98.3:42057)
I20260620 09:17:21.670717 10596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42415
I20260620 09:17:21.679193 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:17:21.685318 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.680403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.0.98.62:41775)
0620 09:17:21.680467 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:21.680468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:21.680541 (+ 73us) server_negotiation.cc:407] Connection header received
0620 09:17:21.680583 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:21.680584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:21.680592 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:21.680641 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:17:21.681109 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.681557 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.682230 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.682450 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.682502 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.683442 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:17:21.683444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:17:21.683444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:17:21.683450 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:17:21.684514 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:21.684788 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:21.684789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:21.684789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:21.684831 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:17:21.684946 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:17:21.684947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:17:21.684947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:17:21.685068 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:17:21.685143 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:21.685209 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 09:17:21.685260 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:17:21.686575 10596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:17:21.688069 10596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:17:21.693889 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.690105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.0.98.2:33417)
0620 09:17:21.690412 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:21.690416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:21.690423 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:17:21.690454 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:21.690456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:21.690468 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:21.690534 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:17:21.690716 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.691503 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.692830 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.693524 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.693583 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.693656 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:21.693766 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 09:17:21.693807 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":87,"threads_started":1}
I20260620 09:17:21.693879 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.689930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33417 (local address 127.0.0.1:51016)
0620 09:17:21.690086 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:21.690122 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:21.690191 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:21.690551 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:21.690554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:21.690556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:17:21.690620 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:21.690622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.691600 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.691601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:21.692734 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:21.692737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.693591 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.693592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:21.693669 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.693683 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 09:17:21.693745 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 09:17:21.693798 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"spinlock_wait_cycles":384,"thread_start_us":59,"threads_started":1}
I20260620 09:17:21.694176 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.690120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36533 (local address 127.0.0.1:38530)
0620 09:17:21.690519 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:21.690528 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:21.690580 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:21.690804 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:21.690804 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:21.690806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:17:21.690853 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:21.690854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.692204 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.692205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:21.693079 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:21.693081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.693947 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.693948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:21.694006 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.694010 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:17:21.694057 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:17:21.694105 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":34,"thread_start_us":80,"threads_started":1}
I20260620 09:17:21.694397 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.689931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42057 (local address 127.0.0.1:60208)
0620 09:17:21.690248 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:21.690258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:21.690309 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:21.690854 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:21.690854 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:21.690857 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:17:21.690890 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:21.690891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.692187 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.692188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:21.693290 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:21.693292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.694127 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.694127 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:21.694187 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.694190 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:17:21.694238 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 09:17:21.694336 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":30,"thread_start_us":52,"threads_started":1}
I20260620 09:17:21.694415 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.690351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.0.98.1:36533)
0620 09:17:21.690506 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:21.690510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:21.690569 (+ 59us) server_negotiation.cc:407] Connection header received
0620 09:17:21.690640 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:21.690642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:21.690657 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:21.690725 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:17:21.690923 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.692089 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.693151 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.693794 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.693860 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.693914 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:21.694299 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 09:17:21.694340 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260620 09:17:21.694794 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:21.690000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.0.98.3:42057)
0620 09:17:21.690710 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:21.690714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:21.690722 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:17:21.690754 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:21.690756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:21.690769 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:21.690954 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:17:21.691019 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.692089 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.693372 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:21.694035 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:21.694120 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:21.694184 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:21.694412 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 09:17:21.694733 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":93,"threads_started":1}
I20260620 09:17:21.694847 10855 tablet_service.cc:1467] Processing CreateTablet for tablet 1f955e24306845c0aec5bd122aa93bb2 (DEFAULT_TABLE table=test-table [id=f138057c75cb4e53891451d19a0a4459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:17:21.695009 10721 tablet_service.cc:1467] Processing CreateTablet for tablet 1f955e24306845c0aec5bd122aa93bb2 (DEFAULT_TABLE table=test-table [id=f138057c75cb4e53891451d19a0a4459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:17:21.695204 10855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f955e24306845c0aec5bd122aa93bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:17:21.695307 10721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f955e24306845c0aec5bd122aa93bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:17:21.695423 10989 tablet_service.cc:1467] Processing CreateTablet for tablet 1f955e24306845c0aec5bd122aa93bb2 (DEFAULT_TABLE table=test-table [id=f138057c75cb4e53891451d19a0a4459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:17:21.695745 10989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f955e24306845c0aec5bd122aa93bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:17:21.698355 11069 tablet_bootstrap.cc:492] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df: Bootstrap starting.
I20260620 09:17:21.698767 11071 tablet_bootstrap.cc:492] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27: Bootstrap starting.
I20260620 09:17:21.699030 11069 tablet_bootstrap.cc:654] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df: Neither blocks nor log segments found. Creating new log.
I20260620 09:17:21.699317 11071 tablet_bootstrap.cc:654] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27: Neither blocks nor log segments found. Creating new log.
I20260620 09:17:21.699451 11069 log.cc:824] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df: Log is configured to *not* fsync() on all Append() calls
I20260620 09:17:21.699775 11070 tablet_bootstrap.cc:492] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844: Bootstrap starting.
I20260620 09:17:21.699844 11071 log.cc:824] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27: Log is configured to *not* fsync() on all Append() calls
I20260620 09:17:21.700152 11069 tablet_bootstrap.cc:492] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df: No bootstrap required, opened a new log
I20260620 09:17:21.700225 11069 ts_tablet_manager.cc:1397] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:17:21.700429 11070 tablet_bootstrap.cc:654] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844: Neither blocks nor log segments found. Creating new log.
I20260620 09:17:21.700639 11071 tablet_bootstrap.cc:492] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27: No bootstrap required, opened a new log
I20260620 09:17:21.700709 11071 ts_tablet_manager.cc:1397] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:17:21.700932 11070 log.cc:824] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844: Log is configured to *not* fsync() on all Append() calls
I20260620 09:17:21.701819 11070 tablet_bootstrap.cc:492] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844: No bootstrap required, opened a new log
I20260620 09:17:21.701887 11070 ts_tablet_manager.cc:1397] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:17:21.701896 11069 raft_consensus.cc:348] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc22423bbdd847eba6fa8a9141a8db27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33417 } } peers { permanent_uuid: "9dbb55e4483045f18cb9f245518f79df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42057 } } peers { permanent_uuid: "2b78ee6ff99f4a7298a3c90e60360844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36533 } }
I20260620 09:17:21.702056 11069 raft_consensus.cc:374] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:17:21.702095 11069 raft_consensus.cc:729] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbb55e4483045f18cb9f245518f79df, State: Initialized, Role: FOLLOWER
I20260620 09:17:21.702217 11069 consensus_queue.cc:260] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc22423bbdd847eba6fa8a9141a8db27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33417 } } peers { permanent_uuid: "9dbb55e4483045f18cb9f245518f79df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42057 } } peers { permanent_uuid: "2b78ee6ff99f4a7298a3c90e60360844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36533 } }
I20260620 09:17:21.702452 11069 ts_tablet_manager.cc:1428] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 09:17:21.702569 11055 heartbeater.cc:503] Master 127.0.98.62:41775 was elected leader, sending a full tablet report...
I20260620 09:17:21.702469 11071 raft_consensus.cc:348] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc22423bbdd847eba6fa8a9141a8db27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33417 } } peers { permanent_uuid: "9dbb55e4483045f18cb9f245518f79df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42057 } } peers { permanent_uuid: "2b78ee6ff99f4a7298a3c90e60360844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36533 } }
I20260620 09:17:21.702643 11071 raft_consensus.cc:374] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:17:21.702683 11071 raft_consensus.cc:729] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc22423bbdd847eba6fa8a9141a8db27, State: Initialized, Role: FOLLOWER
I20260620 09:17:21.702793 11071 consensus_queue.cc:260] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc22423bbdd847eba6fa8a9141a8db27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33417 } } peers { permanent_uuid: "9dbb55e4483045f18cb9f245518f79df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42057 } } peers { permanent_uuid: "2b78ee6ff99f4a7298a3c90e60360844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36533 } }
I20260620 09:17:21.703043 11071 ts_tablet_manager.cc:1428] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:17:21.703048 10921 heartbeater.cc:503] Master 127.0.98.62:41775 was elected leader, sending a full tablet report...
I20260620 09:17:21.703764 11070 raft_consensus.cc:348] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc22423bbdd847eba6fa8a9141a8db27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33417 } } peers { permanent_uuid: "9dbb55e4483045f18cb9f245518f79df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42057 } } peers { permanent_uuid: "2b78ee6ff99f4a7298a3c90e60360844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36533 } }
I20260620 09:17:21.703943 11070 raft_consensus.cc:374] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:17:21.703974 11070 raft_consensus.cc:729] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b78ee6ff99f4a7298a3c90e60360844, State: Initialized, Role: FOLLOWER
I20260620 09:17:21.704046 11070 consensus_queue.cc:260] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc22423bbdd847eba6fa8a9141a8db27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33417 } } peers { permanent_uuid: "9dbb55e4483045f18cb9f245518f79df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42057 } } peers { permanent_uuid: "2b78ee6ff99f4a7298a3c90e60360844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36533 } }
I20260620 09:17:21.704252 11070 ts_tablet_manager.cc:1428] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:17:21.704300 10787 heartbeater.cc:503] Master 127.0.98.62:41775 was elected leader, sending a full tablet report...
W20260620 09:17:21.767607 10922 tablet.cc:2369] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:17:21.872074 10788 tablet.cc:2369] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:17:21.911449 11056 tablet.cc:2369] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:17:22.109242 11077 raft_consensus.cc:482] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:17:22.109377 11077 raft_consensus.cc:504] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc22423bbdd847eba6fa8a9141a8db27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33417 } } peers { permanent_uuid: "9dbb55e4483045f18cb9f245518f79df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42057 } } peers { permanent_uuid: "2b78ee6ff99f4a7298a3c90e60360844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36533 } }
I20260620 09:17:22.109730 11077 leader_election.cc:288] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc22423bbdd847eba6fa8a9141a8db27 (127.0.98.2:33417), 9dbb55e4483045f18cb9f245518f79df (127.0.98.3:42057)
I20260620 09:17:22.113025 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:22.110003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54043 (local address 127.0.98.2:33417)
0620 09:17:22.110060 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:22.110063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:22.110232 (+ 169us) server_negotiation.cc:407] Connection header received
0620 09:17:22.110277 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:22.110278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:22.110290 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:22.110330 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:17:22.110489 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:22.110966 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:22.112357 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:22.112712 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:22.112763 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:22.112772 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:22.112915 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 09:17:22.112954 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 09:17:22.113051 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:22.110041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46743 (local address 127.0.98.3:42057)
0620 09:17:22.110078 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:22.110080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:22.110174 (+ 94us) server_negotiation.cc:407] Connection header received
0620 09:17:22.110231 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:22.110232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:22.110243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:22.110297 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:17:22.110458 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:22.110960 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:22.112374 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:22.112745 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:22.112810 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:22.112820 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:22.112953 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 09:17:22.112990 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:17:22.113065 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:22.109947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42057 (local address 127.0.98.1:46743)
0620 09:17:22.110100 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:22.110110 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:22.110179 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:22.110306 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:22.110307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:22.110309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:17:22.110391 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:22.110393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:22.111046 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:22.111047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:22.112292 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:22.112295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:22.112824 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:22.112824 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:22.112882 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:22.112886 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:17:22.112931 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:17:22.112967 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260620 09:17:22.113066 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:22.109947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33417 (local address 127.0.98.1:54043)
0620 09:17:22.110134 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:22.110144 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:17:22.110228 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:17:22.110343 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:17:22.110344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:17:22.110346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:17:22.110423 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:22.110425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:22.111042 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:22.111043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:22.112280 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:17:22.112282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:17:22.112786 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:17:22.112787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:17:22.112851 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:22.112855 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:17:22.112902 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:17:22.112938 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":86,"thread_start_us":51,"threads_started":1}
I20260620 09:17:22.113363 11009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f955e24306845c0aec5bd122aa93bb2" candidate_uuid: "2b78ee6ff99f4a7298a3c90e60360844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dbb55e4483045f18cb9f245518f79df" is_pre_election: true
I20260620 09:17:22.113504 11009 raft_consensus.cc:2454] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b78ee6ff99f4a7298a3c90e60360844 in term 0.
I20260620 09:17:22.113502 10875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f955e24306845c0aec5bd122aa93bb2" candidate_uuid: "2b78ee6ff99f4a7298a3c90e60360844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc22423bbdd847eba6fa8a9141a8db27" is_pre_election: true
I20260620 09:17:22.113646 10875 raft_consensus.cc:2454] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b78ee6ff99f4a7298a3c90e60360844 in term 0.
I20260620 09:17:22.113744 10675 leader_election.cc:302] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b78ee6ff99f4a7298a3c90e60360844, 9dbb55e4483045f18cb9f245518f79df; no voters:
I20260620 09:17:22.113844 11077 raft_consensus.cc:2781] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:17:22.113894 11077 raft_consensus.cc:482] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:17:22.113926 11077 raft_consensus.cc:3037] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:17:22.114595 11077 raft_consensus.cc:504] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc22423bbdd847eba6fa8a9141a8db27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33417 } } peers { permanent_uuid: "9dbb55e4483045f18cb9f245518f79df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42057 } } peers { permanent_uuid: "2b78ee6ff99f4a7298a3c90e60360844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36533 } }
I20260620 09:17:22.114722 11077 leader_election.cc:288] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [CANDIDATE]: Term 1 election: Requested vote from peers bc22423bbdd847eba6fa8a9141a8db27 (127.0.98.2:33417), 9dbb55e4483045f18cb9f245518f79df (127.0.98.3:42057)
I20260620 09:17:22.114850 11009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f955e24306845c0aec5bd122aa93bb2" candidate_uuid: "2b78ee6ff99f4a7298a3c90e60360844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dbb55e4483045f18cb9f245518f79df"
I20260620 09:17:22.114861 10875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f955e24306845c0aec5bd122aa93bb2" candidate_uuid: "2b78ee6ff99f4a7298a3c90e60360844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc22423bbdd847eba6fa8a9141a8db27"
I20260620 09:17:22.114920 10875 raft_consensus.cc:3037] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:17:22.114919 11009 raft_consensus.cc:3037] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:17:22.115546 11009 raft_consensus.cc:2454] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b78ee6ff99f4a7298a3c90e60360844 in term 1.
I20260620 09:17:22.115645 10875 raft_consensus.cc:2454] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b78ee6ff99f4a7298a3c90e60360844 in term 1.
I20260620 09:17:22.115705 10675 leader_election.cc:302] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b78ee6ff99f4a7298a3c90e60360844, 9dbb55e4483045f18cb9f245518f79df; no voters:
I20260620 09:17:22.115808 11077 raft_consensus.cc:2781] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:17:22.115950 11077 raft_consensus.cc:686] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 1 LEADER]: Becoming Leader. State: Replica: 2b78ee6ff99f4a7298a3c90e60360844, State: Running, Role: LEADER
I20260620 09:17:22.116034 11077 consensus_queue.cc:237] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc22423bbdd847eba6fa8a9141a8db27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33417 } } peers { permanent_uuid: "9dbb55e4483045f18cb9f245518f79df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42057 } } peers { permanent_uuid: "2b78ee6ff99f4a7298a3c90e60360844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36533 } }
I20260620 09:17:22.116781 10596 catalog_manager.cc:5495] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b78ee6ff99f4a7298a3c90e60360844 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2b78ee6ff99f4a7298a3c90e60360844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc22423bbdd847eba6fa8a9141a8db27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dbb55e4483045f18cb9f245518f79df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b78ee6ff99f4a7298a3c90e60360844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36533 } health_report { overall_health: HEALTHY } } }
I20260620 09:17:22.131664 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:22.129561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.0.98.1:36533)
0620 09:17:22.129601 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 09:17:22.129603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:17:22.129635 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:17:22.129692 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:17:22.129692 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:17:22.129707 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:17:22.129765 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:17:22.129866 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:22.130306 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:22.131054 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:17:22.131238 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:17:22.131288 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:17:22.131389 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:17:22.131477 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:17:22.131531 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:17:22.131566 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 09:17:22.131580 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:17:22.133170 11009 raft_consensus.cc:1261] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df [term 1 FOLLOWER]: Refusing update from remote peer 2b78ee6ff99f4a7298a3c90e60360844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:17:22.133224 10875 raft_consensus.cc:1261] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27 [term 1 FOLLOWER]: Refusing update from remote peer 2b78ee6ff99f4a7298a3c90e60360844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:17:22.133493 11077 consensus_queue.cc:1035] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dbb55e4483045f18cb9f245518f79df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:17:22.133638 11080 consensus_queue.cc:1035] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc22423bbdd847eba6fa8a9141a8db27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:17:22.136171 11084 mvcc.cc:205] Tried to move back new op lower bound from 7298855084575277056 to 7298855084507897856. Current Snapshot: MvccSnapshot[applied={T|T < 7298855084575277056}]
I20260620 09:17:22.136627 11086 mvcc.cc:205] Tried to move back new op lower bound from 7298855084575277056 to 7298855084507897856. Current Snapshot: MvccSnapshot[applied={T|T < 7298855084575277056}]
I20260620 09:17:22.138358 10596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:17:22.138447 10596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:17:22.139089 10596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b21a2d76407a45afbc7074ad72fa2fa8: Sending DeleteTablet for 3 replicas of tablet 1f955e24306845c0aec5bd122aa93bb2
I20260620 09:17:22.139397 10989 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f955e24306845c0aec5bd122aa93bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:17:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60208
I20260620 09:17:22.139390 10855 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f955e24306845c0aec5bd122aa93bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:17:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51016
I20260620 09:17:22.139640 11093 tablet_replica.cc:330] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27: stopping tablet replica
I20260620 09:17:22.139766 11093 raft_consensus.cc:2229] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:17:22.139855 10721 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f955e24306845c0aec5bd122aa93bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:17:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38530
I20260620 09:17:22.139940 11093 raft_consensus.cc:2258] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:17:22.140085 11094 tablet_replica.cc:330] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844: stopping tablet replica
I20260620 09:17:22.140192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 10658
I20260620 09:17:22.140213 11094 raft_consensus.cc:2229] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:17:22.140317 11094 raft_consensus.cc:2258] T 1f955e24306845c0aec5bd122aa93bb2 P 2b78ee6ff99f4a7298a3c90e60360844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:17:22.140710 11093 ts_tablet_manager.cc:1905] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:17:22.141852 11092 tablet_replica.cc:330] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df: stopping tablet replica
I20260620 09:17:22.141916 11092 raft_consensus.cc:2229] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:17:22.141974 11092 raft_consensus.cc:2258] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:17:22.141957 11093 ts_tablet_manager.cc:1918] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:17:22.142027 11093 log.cc:1196] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/1f955e24306845c0aec5bd122aa93bb2
I20260620 09:17:22.142270 11093 ts_tablet_manager.cc:1939] T 1f955e24306845c0aec5bd122aa93bb2 P bc22423bbdd847eba6fa8a9141a8db27: Deleting consensus metadata
I20260620 09:17:22.142292 11092 ts_tablet_manager.cc:1905] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:17:22.142642 10580 catalog_manager.cc:4841] TS bc22423bbdd847eba6fa8a9141a8db27 (127.0.98.2:33417): tablet 1f955e24306845c0aec5bd122aa93bb2 (table test-table [id=f138057c75cb4e53891451d19a0a4459]) successfully deleted
I20260620 09:17:22.143340 11092 ts_tablet_manager.cc:1918] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:17:22.143391 11092 log.cc:1196] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/1f955e24306845c0aec5bd122aa93bb2
I20260620 09:17:22.143558 11092 ts_tablet_manager.cc:1939] T 1f955e24306845c0aec5bd122aa93bb2 P 9dbb55e4483045f18cb9f245518f79df: Deleting consensus metadata
I20260620 09:17:22.143878 10581 catalog_manager.cc:4841] TS 9dbb55e4483045f18cb9f245518f79df (127.0.98.3:42057): tablet 1f955e24306845c0aec5bd122aa93bb2 (table test-table [id=f138057c75cb4e53891451d19a0a4459]) successfully deleted
W20260620 09:17:22.145735 10580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36533 (error 108)
I20260620 09:17:22.145911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 10792
I20260620 09:17:22.146414 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:17:22.146013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36533 (local address 127.0.0.1:38548)
0620 09:17:22.146350 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 09:17:22.146387 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317}
W20260620 09:17:22.146569 10580 catalog_manager.cc:4584] TS 2b78ee6ff99f4a7298a3c90e60360844 (127.0.98.1:36533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f955e24306845c0aec5bd122aa93bb2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36533: connect: Connection refused (error 111)
I20260620 09:17:22.150506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 10926
I20260620 09:17:22.154438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 10564
2026-06-20T09:17:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2136 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:17:22 dist-test-slave-rq8c krb5kdc[11098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:17:22 dist-test-slave-rq8c krb5kdc[11098](info): set up 2 sockets
Jun 20 09:17:22 dist-test-slave-rq8c krb5kdc[11098](info): commencing operation
krb5kdc: starting...
W20260620 09:17:23.100589 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:17:23 dist-test-slave-rq8c krb5kdc[11098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:17:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:17:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 09:17:23.241905 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7HdOzb/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/postgres -l logfile start
2026-06-20 09:17:25.460 UTC [11128] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 09:17:25.460 UTC [11128] LOG: listening on IPv4 address "127.0.98.20", port 42327
2026-06-20 09:17:25.463 UTC [11128] LOG: listening on Unix socket "/tmp/.s.PGSQL.42327"
2026-06-20 09:17:25.483 UTC [11131] LOG: database system was shut down at 2026-06-20 09:17:24 UTC
2026-06-20 09:17:25.486 UTC [11128] LOG: database system is ready to accept connections
I20260620 09:17:26.360143 392 mini_postgres.cc:96] Postgres bound to 42327
2026-06-20 09:17:26.365 UTC [11139] FATAL: database "slave" does not exist
127.0.98.20:42327 - accepting connections
I20260620 09:17:26.366219 392 mini_ranger.cc:161] Starting Ranger...
I20260620 09:17:26.379859 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 09:17:26.822104 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 09:17:26.822163 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-admin
2026-06-20 09:17:27,089 [I] DB FLAVOR :POSTGRES
2026-06-20 09:17:27,090 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:17:27,090 [I] Checking connection..
2026-06-20 09:17:27,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 09:17:27,344 [I] Checking connection passed.
2026-06-20 09:17:27,344 [I] --------- Verifying version history table ---------
2026-06-20 09:17:27,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:17:27,580 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:17:27,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:17:27,817 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:17:27,817 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 09:17:27,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 09:17:28.055 UTC [11203] WARNING: there is no transaction in progress
2026-06-20 09:17:28,060 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 09:17:28,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:17:28,313 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 09:17:28,313 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:17:28,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 09:17:28,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 09:17:28,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rq8c.c.gcp-upstream.internal','N') ;"
2026-06-20 09:17:29,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:17:29,266 [I] Table x_portal_user does not exist in database ranger
2026-06-20 09:17:29,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:17:29,499 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 09:17:29,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:17:29,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:17:29,975 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 09:17:29,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 09:17:30.895 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:30.909 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:30.925 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:30.940 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:30.954 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:30.970 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:31.023 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:31.029 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:31.042 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:31.051 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:31.061 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:31.073 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:31.083 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:31.088 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:31.094 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:31.100 UTC [11311] WARNING: there is no transaction in progress
2026-06-20 09:17:31,655 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 09:17:31,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 09:17:31,885 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 09:17:31,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:17:32,111 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 09:17:32,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:17:32,345 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 09:17:32,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:17:32,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:17:32,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rq8c.c.gcp-upstream.internal';"
2026-06-20 09:17:33,067 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 09:17:33,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 09:17:33,321 [I] DB_PATCHES have already been applied
I20260620 09:17:33.326756 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7HdOzb/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 09:17:33.326818 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 09:17:33.329722 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:46485
Jun 20, 2026 9:17:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:17:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:17:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:17:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:17:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:17:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:17:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 09:17:34.335270 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.969s user 0.000s sys 0.008s
Jun 20, 2026 9:17:54 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:17:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 9:17:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:17:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:17:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:17:55 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 09:17:55.951143 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 09:17:56 dist-test-slave-rq8c krb5kdc[11098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947076, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 09:17:56.023059 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 09:17:56.030869 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 09:17:56.751856 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 09:17:56.751932 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-kms
I20260620 09:17:56.751948 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:42327
2026-06-20 09:17:56,910 [I] DB FLAVOR :POSTGRES
2026-06-20 09:17:56,910 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:17:56,910 [I] Checking connection
2026-06-20 09:17:56,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 09:17:57,164 [I] connection success
2026-06-20 09:17:57,164 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 09:17:57,164 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 09:17:57,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 09:17:57,385 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 09:17:57,385 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:17:57,386 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 09:17:57,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 09:17:57,647 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 09:17:58.112612 392 mini_ranger_kms.cc:310] Created kms service
I20260620 09:17:58.304461 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 09:17:58.346312 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 09:17:58 dist-test-slave-rq8c krb5kdc[11098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947076, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:17:58.737466 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 09:17:58.737581 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 09:17:58.737600 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 09:17:58.741170 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11491
I20260620 09:17:58.741315 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:35301
Jun 20, 2026 9:17:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:17:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:17:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:17:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:18:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:18:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 09:18:00.755242 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.732s user 0.007s sys 0.134s
I20260620 09:18:00.755331 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 09:18:00.755393 392 mini_ranger_kms.cc:401] 127.0.98.20:35301/kms/v1/keys
Jun 20 09:18:00 dist-test-slave-rq8c krb5kdc[11098](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781947080, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 09:18:01 dist-test-slave-rq8c krb5kdc[11098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947080, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:18:01 dist-test-slave-rq8c krb5kdc[11098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947080, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:18:03 dist-test-slave-rq8c krb5kdc[11098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:18:03.833968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35593
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35593
--ranger_config_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35301/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:03.910914 11542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:03.911123 11542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:03.911144 11542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:03.912330 11542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:18:03.912380 11542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:03.912393 11542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:03.912405 11542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:18:03.912416 11542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:18:03.913976 11542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:35301/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35593
--ranger_config_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:03.914196 11542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:03.914414 11542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:03.919505 11547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:03.919510 11550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:03.919524 11548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:03.919771 11542 server_base.cc:1034] running on GCE node
I20260620 09:18:03.920066 11542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:03.920377 11542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:03.921511 11542 hybrid_clock.cc:648] HybridClock initialized: now 1781947083921498 us; error 31 us; skew 500 ppm
Jun 20 09:18:03 dist-test-slave-rq8c krb5kdc[11098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:03.924007 11542 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:18:03.924469 11542 webserver.cc:460] Webserver started at http://127.0.98.62:36923/ using document root <none> and password file <none>
I20260620 09:18:03.924664 11542 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:03.924700 11542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:03.924818 11542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:18:03 dist-test-slave-rq8c krb5kdc[11098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:03.985666 11542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "a34c219bd6ae43f9a46f1be160e03c30"
format_stamp: "Formatted at 2026-06-20 09:18:03 on dist-test-slave-rq8c"
server_key: "94c799cfc76f473ccd22bf8cd73cecf8"
server_key_iv: "43dd9f0dd84aaa116c24ba0c9693f411"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:03.986034 11542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "a34c219bd6ae43f9a46f1be160e03c30"
format_stamp: "Formatted at 2026-06-20 09:18:03 on dist-test-slave-rq8c"
server_key: "94c799cfc76f473ccd22bf8cd73cecf8"
server_key_iv: "43dd9f0dd84aaa116c24ba0c9693f411"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:03.987566 11542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:18:04.001286 11557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:04.001525 11542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:18:04.001623 11542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "a34c219bd6ae43f9a46f1be160e03c30"
format_stamp: "Formatted at 2026-06-20 09:18:03 on dist-test-slave-rq8c"
server_key: "94c799cfc76f473ccd22bf8cd73cecf8"
server_key_iv: "43dd9f0dd84aaa116c24ba0c9693f411"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:04.001722 11542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:04.021173 11542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:04.023000 11542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:04.023136 11542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:04.027856 11542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35593
I20260620 09:18:04.027891 11609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35593 every 8 connection(s)
I20260620 09:18:04.028258 11542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:18:04.028870 11610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:04.029111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 11542
I20260620 09:18:04.029217 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:18:04.029414 392 external_mini_cluster.cc:1428] Setting key beedb3e5ed456d16e70895a6fd16c6d2
I20260620 09:18:04.032310 11610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30: Bootstrap starting.
I20260620 09:18:04.032895 11610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:04.033154 11610 log.cc:824] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:04.033917 11610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30: No bootstrap required, opened a new log
I20260620 09:18:04.036211 11610 raft_consensus.cc:348] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34c219bd6ae43f9a46f1be160e03c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260620 09:18:04.036398 11610 raft_consensus.cc:374] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:04.036432 11610 raft_consensus.cc:729] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34c219bd6ae43f9a46f1be160e03c30, State: Initialized, Role: FOLLOWER
I20260620 09:18:04.036623 11610 consensus_queue.cc:260] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34c219bd6ae43f9a46f1be160e03c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260620 09:18:04.036711 11610 raft_consensus.cc:388] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:18:04.036761 11610 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:18:04.036814 11610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 09:18:04 dist-test-slave-rq8c krb5kdc[11098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:04.037364 11610 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34c219bd6ae43f9a46f1be160e03c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260620 09:18:04.037469 11610 leader_election.cc:302] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a34c219bd6ae43f9a46f1be160e03c30; no voters:
I20260620 09:18:04.037642 11610 leader_election.cc:288] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:18:04.037694 11615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:18:04.038005 11610 sys_catalog.cc:564] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:18:04.038223 11615 raft_consensus.cc:686] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [term 1 LEADER]: Becoming Leader. State: Replica: a34c219bd6ae43f9a46f1be160e03c30, State: Running, Role: LEADER
I20260620 09:18:04.038759 11615 consensus_queue.cc:237] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34c219bd6ae43f9a46f1be160e03c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260620 09:18:04.040443 11617 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a34c219bd6ae43f9a46f1be160e03c30. Latest consensus state: current_term: 1 leader_uuid: "a34c219bd6ae43f9a46f1be160e03c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34c219bd6ae43f9a46f1be160e03c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } }
I20260620 09:18:04.040520 11616 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a34c219bd6ae43f9a46f1be160e03c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34c219bd6ae43f9a46f1be160e03c30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } }
I20260620 09:18:04.040536 11617 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:04.040720 11616 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:04.040452 11610 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 09:18:04.040886 11629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:18:04.041378 11629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:18:04.041616 11613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:04.029900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45194 (local address 127.0.98.62:35593)
0620 09:18:04.030091 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:04.030102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:04.030141 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:18:04.030737 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:04.030748 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:04.030817 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:04.030974 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:04.031708 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:04.033149 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:04.034151 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:04.034765 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:04.034879 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:04.037835 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:04.037839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:04.037839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:04.037862 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:04.039726 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:04.040146 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:04.040149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:04.040150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:04.040249 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:04.040727 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:04.040728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:04.040729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:04.041035 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:04.041177 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:04.041298 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 09:18:04.041413 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260620 09:18:04.042922 11629 catalog_manager.cc:1347] Generated new cluster ID: 972ad3701bc04504850f66a8f251e816
I20260620 09:18:04.042970 11629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:18:04.071298 11629 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:18:04.072196 11629 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:18:04.085961 11629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30: Generated new TSK 0
I20260620 09:18:04.086272 11629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:18:04.753628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35301/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:04.839059 11655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:04.839246 11655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:04.839277 11655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:04.840472 11655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:04.840512 11655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:04.840548 11655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:18:04.841969 11655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:35301/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:04.842190 11655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:04.842414 11655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:04.847076 11663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:04.847076 11660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:04.847136 11661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:04.847204 11655 server_base.cc:1034] running on GCE node
I20260620 09:18:04.847497 11655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:04.847783 11655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:04.848928 11655 hybrid_clock.cc:648] HybridClock initialized: now 1781947084848918 us; error 27 us; skew 500 ppm
Jun 20 09:18:04 dist-test-slave-rq8c krb5kdc[11098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:04.851673 11655 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:18:04.852134 11655 webserver.cc:460] Webserver started at http://127.0.98.1:45709/ using document root <none> and password file <none>
I20260620 09:18:04.852361 11655 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:04.852404 11655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:04.852511 11655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:18:04 dist-test-slave-rq8c krb5kdc[11098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:04.867158 11655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2"
format_stamp: "Formatted at 2026-06-20 09:18:04 on dist-test-slave-rq8c"
server_key: "8a9f7e58aa9a49534b7c8f503aed7e34"
server_key_iv: "18c6a638566741c8ba1408532b291494"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:04.867604 11655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2"
format_stamp: "Formatted at 2026-06-20 09:18:04 on dist-test-slave-rq8c"
server_key: "8a9f7e58aa9a49534b7c8f503aed7e34"
server_key_iv: "18c6a638566741c8ba1408532b291494"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:04.869387 11655 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 09:18:04.882794 11670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:04.883351 11655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:18:04.883464 11655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2"
format_stamp: "Formatted at 2026-06-20 09:18:04 on dist-test-slave-rq8c"
server_key: "8a9f7e58aa9a49534b7c8f503aed7e34"
server_key_iv: "18c6a638566741c8ba1408532b291494"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:04.883569 11655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:04.896838 11655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:04.898922 11655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:04.899071 11655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:04.899293 11655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:04.899655 11655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:04.899770 11655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:04.899816 11655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:04.899835 11655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:04.907222 11655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38875
I20260620 09:18:04.907323 11783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38875 every 8 connection(s)
I20260620 09:18:04.907639 11655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:18:04.910015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 11655
I20260620 09:18:04.910112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:18:04.910297 392 external_mini_cluster.cc:1428] Setting key a0b5547280b063796156a57a10c7541e
Jun 20 09:18:04 dist-test-slave-rq8c krb5kdc[11098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:18:04.918363 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:04.908900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34117 (local address 127.0.98.62:35593)
0620 09:18:04.909036 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:04.909038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:04.909131 (+ 93us) server_negotiation.cc:407] Connection header received
0620 09:18:04.909816 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:04.909818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:04.909833 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:04.909919 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:04.910589 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:04.911213 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:04.912277 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:04.912568 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:04.912630 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:04.915041 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:04.915043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:04.915043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:04.915051 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:04.916810 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:04.917256 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:04.917258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:04.917258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:04.917328 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:04.917610 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:04.917611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:04.917611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:04.917763 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:04.917837 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:04.918171 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 09:18:04.918254 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260620 09:18:04.918454 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:04.908836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.1:34117)
0620 09:18:04.909007 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:04.909056 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:04.909678 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:04.909995 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:04.910000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:04.910222 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:04.910499 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:04.910506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:04.911305 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:04.911307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:04.912174 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:04.912177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:04.912655 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:04.912656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:04.912820 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:04.913377 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:04.913387 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:04.914941 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:04.916896 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:04.916902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:04.916904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:04.917170 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:04.917402 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:04.917403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:04.917403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:04.917533 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:04.917834 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:04.917839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:04.918016 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 09:18:04.918133 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 09:18:04.918283 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260620 09:18:04.918943 11784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35593
I20260620 09:18:04.919027 11784 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:04.919184 11784 heartbeater.cc:511] Master 127.0.98.62:35593 requested a full tablet report, sending...
I20260620 09:18:04.919744 11574 ts_manager.cc:194] Registered new tserver with Master: 9dc9bac7497f4bdc9ed63d2bde8969a2 (127.0.98.1:38875)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:18:04.920567 11574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:18:04.932592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35301/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:05.027374 11792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:05.029340 11792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:05.029393 11792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:05.031076 11792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:05.031184 11792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:05.031267 11792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:18:05.033476 11792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:35301/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:05.033929 11792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:05.034251 11792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:05.041112 11799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:05.041383 11800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:05.041595 11802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:05.042498 11792 server_base.cc:1034] running on GCE node
I20260620 09:18:05.042783 11792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:05.043025 11792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:05.044183 11792 hybrid_clock.cc:648] HybridClock initialized: now 1781947085044163 us; error 34 us; skew 500 ppm
Jun 20 09:18:05 dist-test-slave-rq8c krb5kdc[11098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:05.047398 11792 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:18:05.047902 11792 webserver.cc:460] Webserver started at http://127.0.98.2:46373/ using document root <none> and password file <none>
I20260620 09:18:05.048149 11792 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:05.048198 11792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:05.048314 11792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:18:05 dist-test-slave-rq8c krb5kdc[11098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:05.065560 11792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17"
format_stamp: "Formatted at 2026-06-20 09:18:05 on dist-test-slave-rq8c"
server_key: "5ef5601926868e5d16d2c15f5cab9bdb"
server_key_iv: "b24798b9202254188071a703e312626e"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:05.066105 11792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17"
format_stamp: "Formatted at 2026-06-20 09:18:05 on dist-test-slave-rq8c"
server_key: "5ef5601926868e5d16d2c15f5cab9bdb"
server_key_iv: "b24798b9202254188071a703e312626e"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:05.067965 11792 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 09:18:05.084731 11809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:05.084972 11792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 09:18:05.085067 11792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17"
format_stamp: "Formatted at 2026-06-20 09:18:05 on dist-test-slave-rq8c"
server_key: "5ef5601926868e5d16d2c15f5cab9bdb"
server_key_iv: "b24798b9202254188071a703e312626e"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:05.085165 11792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:05.110760 11792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:05.113281 11792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:05.113480 11792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:05.113780 11792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:05.114194 11792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:05.114279 11792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:05.114351 11792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:05.114418 11792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:18:05 dist-test-slave-rq8c krb5kdc[11098](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781947085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:05.121313 11792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34991
I20260620 09:18:05.121852 11792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:18:05.122921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 11792
I20260620 09:18:05.123023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:18:05.123202 392 external_mini_cluster.cc:1428] Setting key 74df4a330caca4773cf8eb757681b1f1
I20260620 09:18:05.128486 11922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34991 every 8 connection(s)
Jun 20 09:18:05 dist-test-slave-rq8c krb5kdc[11098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:18:05.154749 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.129750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59361 (local address 127.0.98.62:35593)
0620 09:18:05.129796 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:05.129799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:05.139983 (+ 10184us) server_negotiation.cc:407] Connection header received
0620 09:18:05.140809 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:05.140811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:05.140825 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:05.140857 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:05.141762 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.142533 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.144029 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.144318 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.144361 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.151183 (+ 6822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:05.151185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:05.151186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:05.151195 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:05.153022 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:05.153587 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:05.153588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:05.153589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:05.153648 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:05.153958 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:05.153959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:05.153959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:05.154106 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:05.154164 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:05.154559 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 09:18:05.154648 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:18:05.155544 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.129446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.2:59361)
0620 09:18:05.139756 (+ 10310us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:05.139837 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:05.140676 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:05.141087 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:05.141092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:05.141356 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:05.141646 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.141653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.142655 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.142656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.143933 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.143936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.144457 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.144457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.144611 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.145227 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:05.145238 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:05.151050 (+ 5812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:05.153127 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:05.153136 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:05.153139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:05.153489 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:05.153714 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:05.153714 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:05.153715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:05.153867 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:05.154199 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:05.154206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:05.154404 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 09:18:05.155120 (+ 716us) client_negotiation.cc:240] Negotiation successful
0620 09:18:05.155303 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10232,"thread_start_us":97,"threads_started":1}
I20260620 09:18:05.156235 11923 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35593
I20260620 09:18:05.156330 11923 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:05.156527 11923 heartbeater.cc:511] Master 127.0.98.62:35593 requested a full tablet report, sending...
I20260620 09:18:05.156961 11574 ts_manager.cc:194] Registered new tserver with Master: 16ef0b8511c14e13a5cc28e8b4bd0f17 (127.0.98.2:34991)
I20260620 09:18:05.157454 11574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:18:05.176689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35301/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:05.296847 11928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:05.297164 11928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:05.297222 11928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:05.298904 11928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:05.299082 11928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:05.299153 11928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:18:05.301256 11928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:35301/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:05.301614 11928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:05.301883 11928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:05.312429 11935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:05.312407 11934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:05.312844 11937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:05.314167 11928 server_base.cc:1034] running on GCE node
I20260620 09:18:05.314379 11928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:05.314626 11928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:05.315768 11928 hybrid_clock.cc:648] HybridClock initialized: now 1781947085315748 us; error 33 us; skew 500 ppm
Jun 20 09:18:05 dist-test-slave-rq8c krb5kdc[11098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:05.319128 11928 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:18:05.319604 11928 webserver.cc:460] Webserver started at http://127.0.98.3:35071/ using document root <none> and password file <none>
I20260620 09:18:05.319948 11928 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:05.319994 11928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:05.320111 11928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:18:05 dist-test-slave-rq8c krb5kdc[11098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:05.345409 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "c25723769f2d4e00b428e3a1548529ce"
format_stamp: "Formatted at 2026-06-20 09:18:05 on dist-test-slave-rq8c"
server_key: "3aec79341caea3a0c72fc388798825c1"
server_key_iv: "f318bc4d276a7ea10e31379428c75578"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:05.345979 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c25723769f2d4e00b428e3a1548529ce"
format_stamp: "Formatted at 2026-06-20 09:18:05 on dist-test-slave-rq8c"
server_key: "3aec79341caea3a0c72fc388798825c1"
server_key_iv: "f318bc4d276a7ea10e31379428c75578"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:05.348552 11928 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:18:05.362135 11944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:05.362367 11928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:18:05.362442 11928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "c25723769f2d4e00b428e3a1548529ce"
format_stamp: "Formatted at 2026-06-20 09:18:05 on dist-test-slave-rq8c"
server_key: "3aec79341caea3a0c72fc388798825c1"
server_key_iv: "f318bc4d276a7ea10e31379428c75578"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:05.362529 11928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:05.379348 11928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:05.381699 11928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:05.381906 11928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:05.382225 11928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:05.382618 11928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:05.382701 11928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:05.382768 11928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:05.382829 11928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:05.389638 11928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44733
I20260620 09:18:05.390089 11928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:18:05.390697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 11928
I20260620 09:18:05.390786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:18:05.390969 392 external_mini_cluster.cc:1428] Setting key 10c6531e3684898aed05e9a253a20feb
I20260620 09:18:05.392841 12057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44733 every 8 connection(s)
Jun 20 09:18:05 dist-test-slave-rq8c krb5kdc[11098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:05.403985 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.394191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60477 (local address 127.0.98.62:35593)
0620 09:18:05.394230 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:05.394232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:05.394417 (+ 185us) server_negotiation.cc:407] Connection header received
0620 09:18:05.395049 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:05.395050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:05.395062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:05.395088 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:05.395865 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.396575 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.397964 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.398241 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.398277 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.400625 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:05.400627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:05.400627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:05.400634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:05.402110 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:05.403008 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:05.403010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:05.403010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:05.403062 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:05.403297 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:05.403298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:05.403298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:05.403414 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:05.403450 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:05.403806 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 09:18:05.403894 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:18:05.404731 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.393938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.3:60477)
0620 09:18:05.394309 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:05.394351 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:05.394942 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:05.395256 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:05.395260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:05.395470 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:05.395756 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.395762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.396678 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.396679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.397873 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.397875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.398354 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.398355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.398481 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.399013 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:05.399022 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:05.400531 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:05.402653 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:05.402658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:05.402661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:05.402932 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:05.403124 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:05.403124 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:05.403124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:05.403236 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:05.403489 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:05.403494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:05.403671 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 09:18:05.404322 (+ 651us) client_negotiation.cc:240] Negotiation successful
0620 09:18:05.404468 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":80,"threads_started":1}
I20260620 09:18:05.405287 12058 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35593
I20260620 09:18:05.405383 12058 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:05.405547 12058 heartbeater.cc:511] Master 127.0.98.62:35593 requested a full tablet report, sending...
I20260620 09:18:05.405954 11574 ts_manager.cc:194] Registered new tserver with Master: c25723769f2d4e00b428e3a1548529ce (127.0.98.3:44733)
I20260620 09:18:05.406412 11574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60477
I20260620 09:18:05.413323 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:18:05.420362 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.414671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.0.98.62:35593)
0620 09:18:05.414710 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:05.414712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:05.414826 (+ 114us) server_negotiation.cc:407] Connection header received
0620 09:18:05.414872 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:05.414872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:05.414888 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:05.414930 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:05.415450 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.415986 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.416701 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.416901 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.416954 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.418057 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:05.418058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:05.418058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:05.418065 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:05.419379 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:05.419699 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:05.419700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:05.419700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:05.419753 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:05.419910 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:05.419911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:05.419911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:05.420012 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:05.420095 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:05.420201 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 09:18:05.420270 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:18:05.422003 11574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:18:05.423578 11574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:18:05.431085 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.425835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38875 (local address 127.0.0.1:52756)
0620 09:18:05.426241 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:05.426284 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:05.426345 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:05.426700 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:05.426703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:05.426706 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:05.426770 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.426771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.428581 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.428582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.429815 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.429817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.430799 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.430800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.430873 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.430887 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 09:18:05.430951 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 09:18:05.430999 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":79,"threads_started":1}
I20260620 09:18:05.432192 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.425835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44733 (local address 127.0.0.1:60048)
0620 09:18:05.427128 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:05.427138 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:05.427185 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:05.427420 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:05.427420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:05.427422 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:05.427466 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.427467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.429540 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.429541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.430629 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.430632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.431971 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.431971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.432029 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.432033 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:18:05.432077 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 09:18:05.432116 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260620 09:18:05.432543 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.426325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.0.98.3:44733)
0620 09:18:05.427000 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:05.427004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:05.427233 (+ 229us) server_negotiation.cc:407] Connection header received
0620 09:18:05.427272 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:05.427274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:05.427288 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:05.427346 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:05.427532 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.429456 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.430742 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.431781 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.431836 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.431939 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:05.432424 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 09:18:05.432465 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":100,"threads_started":1}
I20260620 09:18:05.433429 11992 tablet_service.cc:1467] Processing CreateTablet for tablet eea4cefae3284876a231b8a5eb2a8c0b (DEFAULT_TABLE table=test-table [id=6c252ef90536422195794571f2270727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:18:05.433779 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea4cefae3284876a231b8a5eb2a8c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:05.435109 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.426086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52756 (local address 127.0.98.1:38875)
0620 09:18:05.426483 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:05.426489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:05.426499 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:18:05.426532 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:05.426534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:05.426551 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:05.426623 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:05.426841 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.428491 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.429898 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.430651 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.430707 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.430771 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:05.434979 (+ 4208us) server_negotiation.cc:299] Negotiation successful
0620 09:18:05.435039 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":55,"threads_started":1}
I20260620 09:18:05.435766 11718 tablet_service.cc:1467] Processing CreateTablet for tablet eea4cefae3284876a231b8a5eb2a8c0b (DEFAULT_TABLE table=test-table [id=6c252ef90536422195794571f2270727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:18:05.436121 11718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea4cefae3284876a231b8a5eb2a8c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:05.438028 12076 tablet_bootstrap.cc:492] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce: Bootstrap starting.
I20260620 09:18:05.438958 12076 tablet_bootstrap.cc:654] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:05.439241 12076 log.cc:824] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:05.439816 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.426048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34991 (local address 127.0.0.1:55292)
0620 09:18:05.431486 (+ 5438us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:05.431495 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:05.431528 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:05.431806 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:05.431807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:05.431809 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:05.431837 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.431838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.433021 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.433022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.433984 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.433987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.439546 (+ 5559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.439547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.439641 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.439647 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:18:05.439696 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 09:18:05.439746 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5429,"thread_start_us":57,"threads_started":1}
I20260620 09:18:05.440212 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.426519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.0.98.2:34991)
0620 09:18:05.427079 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:05.427083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:05.431582 (+ 4499us) server_negotiation.cc:407] Connection header received
0620 09:18:05.431646 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:05.431649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:05.431665 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:05.431727 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:05.431921 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.432942 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.434092 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.434784 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.434851 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.434926 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:05.440072 (+ 5146us) server_negotiation.cc:299] Negotiation successful
0620 09:18:05.440128 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":86,"threads_started":1}
I20260620 09:18:05.440393 12076 tablet_bootstrap.cc:492] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce: No bootstrap required, opened a new log
I20260620 09:18:05.440457 12076 ts_tablet_manager.cc:1397] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:18:05.440821 12077 tablet_bootstrap.cc:492] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2: Bootstrap starting.
I20260620 09:18:05.442334 12076 raft_consensus.cc:348] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25723769f2d4e00b428e3a1548529ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34991 } } peers { permanent_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38875 } }
I20260620 09:18:05.442498 12076 raft_consensus.cc:374] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:05.442533 12076 raft_consensus.cc:729] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25723769f2d4e00b428e3a1548529ce, State: Initialized, Role: FOLLOWER
I20260620 09:18:05.442647 12076 consensus_queue.cc:260] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25723769f2d4e00b428e3a1548529ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34991 } } peers { permanent_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38875 } }
I20260620 09:18:05.442916 12076 ts_tablet_manager.cc:1428] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:18:05.450659 12058 heartbeater.cc:503] Master 127.0.98.62:35593 was elected leader, sending a full tablet report...
I20260620 09:18:05.443487 11857 tablet_service.cc:1467] Processing CreateTablet for tablet eea4cefae3284876a231b8a5eb2a8c0b (DEFAULT_TABLE table=test-table [id=6c252ef90536422195794571f2270727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:18:05.452509 11857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea4cefae3284876a231b8a5eb2a8c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:05.456347 12077 tablet_bootstrap.cc:654] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:05.457298 12077 log.cc:824] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:05.458144 12077 tablet_bootstrap.cc:492] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2: No bootstrap required, opened a new log
I20260620 09:18:05.458226 12077 ts_tablet_manager.cc:1397] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260620 09:18:05.460695 12077 raft_consensus.cc:348] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25723769f2d4e00b428e3a1548529ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34991 } } peers { permanent_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38875 } }
I20260620 09:18:05.460866 12077 raft_consensus.cc:374] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:05.460896 12077 raft_consensus.cc:729] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc9bac7497f4bdc9ed63d2bde8969a2, State: Initialized, Role: FOLLOWER
I20260620 09:18:05.461601 12077 consensus_queue.cc:260] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25723769f2d4e00b428e3a1548529ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34991 } } peers { permanent_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38875 } }
I20260620 09:18:05.461948 12077 ts_tablet_manager.cc:1428] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 09:18:05.462106 11784 heartbeater.cc:503] Master 127.0.98.62:35593 was elected leader, sending a full tablet report...
I20260620 09:18:05.462702 12081 tablet_bootstrap.cc:492] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17: Bootstrap starting.
I20260620 09:18:05.463800 12081 tablet_bootstrap.cc:654] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:05.464150 12081 log.cc:824] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:05.464896 12081 tablet_bootstrap.cc:492] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17: No bootstrap required, opened a new log
I20260620 09:18:05.464958 12081 ts_tablet_manager.cc:1397] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:18:05.466975 12081 raft_consensus.cc:348] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25723769f2d4e00b428e3a1548529ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34991 } } peers { permanent_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38875 } }
I20260620 09:18:05.467141 12081 raft_consensus.cc:374] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:05.467170 12081 raft_consensus.cc:729] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ef0b8511c14e13a5cc28e8b4bd0f17, State: Initialized, Role: FOLLOWER
I20260620 09:18:05.467289 12081 consensus_queue.cc:260] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25723769f2d4e00b428e3a1548529ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34991 } } peers { permanent_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38875 } }
I20260620 09:18:05.467535 12081 ts_tablet_manager.cc:1428] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:18:05.469892 11923 heartbeater.cc:503] Master 127.0.98.62:35593 was elected leader, sending a full tablet report...
Jun 20 09:18:05 dist-test-slave-rq8c krb5kdc[11098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 09:18:05.629333 11924 tablet.cc:2369] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:18:05.643371 12059 tablet.cc:2369] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:18:05.658975 11785 tablet.cc:2369] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:18:05.746301 12084 raft_consensus.cc:482] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:18:05.746454 12084 raft_consensus.cc:504] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25723769f2d4e00b428e3a1548529ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34991 } } peers { permanent_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38875 } }
I20260620 09:18:05.746848 12084 leader_election.cc:288] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c25723769f2d4e00b428e3a1548529ce (127.0.98.3:44733), 9dc9bac7497f4bdc9ed63d2bde8969a2 (127.0.98.1:38875)
I20260620 09:18:05.750874 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.747316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52003 (local address 127.0.98.3:44733)
0620 09:18:05.747358 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:05.747360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:05.747453 (+ 93us) server_negotiation.cc:407] Connection header received
0620 09:18:05.747543 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:05.747543 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:05.747551 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:05.747576 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:05.747807 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.748553 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.749961 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.750523 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.750565 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.750575 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:05.750772 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 09:18:05.750805 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:18:05.750943 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.747238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34635 (local address 127.0.98.1:38875)
0620 09:18:05.747288 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:05.747290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:05.747369 (+ 79us) server_negotiation.cc:407] Connection header received
0620 09:18:05.747471 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:05.747472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:05.747484 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:05.747510 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:05.747746 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.748471 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.749947 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.750547 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.750594 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.750604 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:05.750821 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 09:18:05.750873 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:18:05.751179 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.747046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38875 (local address 127.0.98.2:34635)
0620 09:18:05.747304 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:05.747314 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:05.747396 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:05.747577 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:05.747578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:05.747580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:05.747666 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.747668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.748570 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.748571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.749839 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.749842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.750662 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.750662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.750736 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.750740 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:18:05.751055 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 09:18:05.751101 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260620 09:18:05.751343 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.747128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44733 (local address 127.0.98.2:52003)
0620 09:18:05.747398 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:05.747407 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:05.747476 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:05.747665 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:05.747665 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:05.747667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:05.747736 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.747738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.748654 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.748655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.749864 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:05.749866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.750629 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.750630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:05.750700 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.750704 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:18:05.751206 (+ 502us) client_negotiation.cc:240] Negotiation successful
0620 09:18:05.751259 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":6,"thread_start_us":90,"threads_started":1}
I20260620 09:18:05.751701 11738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea4cefae3284876a231b8a5eb2a8c0b" candidate_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2" is_pre_election: true
I20260620 09:18:05.751852 11738 raft_consensus.cc:2454] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ef0b8511c14e13a5cc28e8b4bd0f17 in term 0.
I20260620 09:18:05.751794 12012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea4cefae3284876a231b8a5eb2a8c0b" candidate_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25723769f2d4e00b428e3a1548529ce" is_pre_election: true
I20260620 09:18:05.751957 12012 raft_consensus.cc:2454] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ef0b8511c14e13a5cc28e8b4bd0f17 in term 0.
I20260620 09:18:05.752170 11810 leader_election.cc:302] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ef0b8511c14e13a5cc28e8b4bd0f17, 9dc9bac7497f4bdc9ed63d2bde8969a2; no voters:
I20260620 09:18:05.752316 12084 raft_consensus.cc:2781] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:18:05.752374 12084 raft_consensus.cc:482] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:18:05.752396 12084 raft_consensus.cc:3037] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:05.753243 12084 raft_consensus.cc:504] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25723769f2d4e00b428e3a1548529ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34991 } } peers { permanent_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38875 } }
I20260620 09:18:05.753374 12084 leader_election.cc:288] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [CANDIDATE]: Term 1 election: Requested vote from peers c25723769f2d4e00b428e3a1548529ce (127.0.98.3:44733), 9dc9bac7497f4bdc9ed63d2bde8969a2 (127.0.98.1:38875)
I20260620 09:18:05.753551 12012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea4cefae3284876a231b8a5eb2a8c0b" candidate_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25723769f2d4e00b428e3a1548529ce"
I20260620 09:18:05.753561 11738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea4cefae3284876a231b8a5eb2a8c0b" candidate_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2"
I20260620 09:18:05.753619 12012 raft_consensus.cc:3037] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:05.753628 11738 raft_consensus.cc:3037] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:05.754426 11738 raft_consensus.cc:2454] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ef0b8511c14e13a5cc28e8b4bd0f17 in term 1.
I20260620 09:18:05.754437 12012 raft_consensus.cc:2454] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ef0b8511c14e13a5cc28e8b4bd0f17 in term 1.
I20260620 09:18:05.754575 11810 leader_election.cc:302] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ef0b8511c14e13a5cc28e8b4bd0f17, 9dc9bac7497f4bdc9ed63d2bde8969a2; no voters:
I20260620 09:18:05.754699 12084 raft_consensus.cc:2781] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:18:05.754869 12084 raft_consensus.cc:686] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 1 LEADER]: Becoming Leader. State: Replica: 16ef0b8511c14e13a5cc28e8b4bd0f17, State: Running, Role: LEADER
I20260620 09:18:05.754962 12084 consensus_queue.cc:237] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25723769f2d4e00b428e3a1548529ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } } peers { permanent_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34991 } } peers { permanent_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38875 } }
I20260620 09:18:05.755769 11573 catalog_manager.cc:5495] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16ef0b8511c14e13a5cc28e8b4bd0f17 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c25723769f2d4e00b428e3a1548529ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16ef0b8511c14e13a5cc28e8b4bd0f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38875 } health_report { overall_health: UNKNOWN } } }
I20260620 09:18:05.791841 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.788550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.0.98.2:34991)
0620 09:18:05.788616 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:05.788619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:05.788628 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:18:05.788670 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:05.788671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:05.788695 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:05.788782 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:18:05.788927 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.789668 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.790797 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:05.791074 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:05.791136 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:05.791341 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:18:05.791474 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:18:05.791548 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:05.791743 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 09:18:05.791763 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:18:05.794409 12012 raft_consensus.cc:1261] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce [term 1 FOLLOWER]: Refusing update from remote peer 16ef0b8511c14e13a5cc28e8b4bd0f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:18:05.794412 11738 raft_consensus.cc:1261] T eea4cefae3284876a231b8a5eb2a8c0b P 9dc9bac7497f4bdc9ed63d2bde8969a2 [term 1 FOLLOWER]: Refusing update from remote peer 16ef0b8511c14e13a5cc28e8b4bd0f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:18:05.794723 12084 consensus_queue.cc:1035] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c25723769f2d4e00b428e3a1548529ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:18:05.794848 12087 consensus_queue.cc:1035] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dc9bac7497f4bdc9ed63d2bde8969a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:18:05.798111 12090 mvcc.cc:205] Tried to move back new op lower bound from 7298855263410892800 to 7298855263252938752. Current Snapshot: MvccSnapshot[applied={T|T < 7298855263410892800}]
I20260620 09:18:05.800041 12092 mvcc.cc:205] Tried to move back new op lower bound from 7298855263410892800 to 7298855263252938752. Current Snapshot: MvccSnapshot[applied={T|T < 7298855263410892800}]
I20260620 09:18:05.802819 11574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:18:05.802912 11574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:18:05.803874 11574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a34c219bd6ae43f9a46f1be160e03c30: Sending DeleteTablet for 3 replicas of tablet eea4cefae3284876a231b8a5eb2a8c0b
I20260620 09:18:05.804203 11718 tablet_service.cc:1514] Processing DeleteTablet for tablet eea4cefae3284876a231b8a5eb2a8c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52756
I20260620 09:18:05.804577 11992 tablet_service.cc:1514] Processing DeleteTablet for tablet eea4cefae3284876a231b8a5eb2a8c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60048
I20260620 09:18:05.804880 11857 tablet_service.cc:1514] Processing DeleteTablet for tablet eea4cefae3284876a231b8a5eb2a8c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55292
I20260620 09:18:05.805452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 11655
I20260620 09:18:05.808995 12101 tablet_replica.cc:330] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17: stopping tablet replica
I20260620 09:18:05.809134 12100 tablet_replica.cc:330] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce: stopping tablet replica
I20260620 09:18:05.809310 12100 raft_consensus.cc:2229] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:18:05.809444 12101 raft_consensus.cc:2229] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:18:05.809608 12101 raft_consensus.cc:2258] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:18:05.809751 12100 raft_consensus.cc:2258] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:18:05.810535 12100 ts_tablet_manager.cc:1905] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:18:05.810910 12101 ts_tablet_manager.cc:1905] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:18:05.813872 12100 ts_tablet_manager.cc:1918] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:18:05.817059 12100 log.cc:1196] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/eea4cefae3284876a231b8a5eb2a8c0b
I20260620 09:18:05.817667 12100 ts_tablet_manager.cc:1939] T eea4cefae3284876a231b8a5eb2a8c0b P c25723769f2d4e00b428e3a1548529ce: Deleting consensus metadata
I20260620 09:18:05.817819 12101 ts_tablet_manager.cc:1918] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:18:05.817924 12101 log.cc:1196] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/eea4cefae3284876a231b8a5eb2a8c0b
I20260620 09:18:05.818174 12101 ts_tablet_manager.cc:1939] T eea4cefae3284876a231b8a5eb2a8c0b P 16ef0b8511c14e13a5cc28e8b4bd0f17: Deleting consensus metadata
I20260620 09:18:05.818635 11559 catalog_manager.cc:4841] TS 16ef0b8511c14e13a5cc28e8b4bd0f17 (127.0.98.2:34991): tablet eea4cefae3284876a231b8a5eb2a8c0b (table test-table [id=6c252ef90536422195794571f2270727]) successfully deleted
I20260620 09:18:05.819113 11559 catalog_manager.cc:4841] TS c25723769f2d4e00b428e3a1548529ce (127.0.98.3:44733): tablet eea4cefae3284876a231b8a5eb2a8c0b (table test-table [id=6c252ef90536422195794571f2270727]) successfully deleted
W20260620 09:18:05.819698 11558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38875 (error 108)
I20260620 09:18:05.820569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 11792
I20260620 09:18:05.820586 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:05.820448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38875 (local address 127.0.0.1:45800)
0620 09:18:05.820496 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:05.820551 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 09:18:05.821449 11558 catalog_manager.cc:4584] TS 9dc9bac7497f4bdc9ed63d2bde8969a2 (127.0.98.1:38875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eea4cefae3284876a231b8a5eb2a8c0b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38875: connect: Connection refused (error 111)
I20260620 09:18:05.827744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 11928
I20260620 09:18:05.832479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 11542
I20260620 09:18:05.838055 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 09:18:06.172353 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 09:18:06.172452 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 09:18:06.506570 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 09:18:06.506 UTC [11128] LOG: received smart shutdown request
2026-06-20 09:18:06.509 UTC [11128] LOG: background worker "logical replication launcher" (PID 11137) exited with exit code 1
2026-06-20 09:18:06.509 UTC [11132] LOG: shutting down
2026-06-20 09:18:06.653 UTC [11128] LOG: database system is shut down
2026-06-20T09:18:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44582 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:18:06 dist-test-slave-rq8c krb5kdc[12111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:18:06 dist-test-slave-rq8c krb5kdc[12111](info): set up 2 sockets
Jun 20 09:18:06 dist-test-slave-rq8c krb5kdc[12111](info): commencing operation
krb5kdc: starting...
W20260620 09:18:07.695356 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:18:07 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:18:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 09:18:07.836061 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7HdOzb/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/postgres -l logfile start
2026-06-20 09:18:10.294 UTC [12141] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 09:18:10.294 UTC [12141] LOG: listening on IPv4 address "127.0.98.20", port 53659
2026-06-20 09:18:10.298 UTC [12141] LOG: listening on Unix socket "/tmp/.s.PGSQL.53659"
2026-06-20 09:18:10.324 UTC [12144] LOG: database system was shut down at 2026-06-20 09:18:08 UTC
2026-06-20 09:18:10.328 UTC [12141] LOG: database system is ready to accept connections
I20260620 09:18:11.191195 392 mini_postgres.cc:96] Postgres bound to 53659
2026-06-20 09:18:11.196 UTC [12152] FATAL: database "slave" does not exist
127.0.98.20:53659 - accepting connections
I20260620 09:18:11.196933 392 mini_ranger.cc:161] Starting Ranger...
I20260620 09:18:11.210453 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 09:18:11.657876 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 09:18:11.657950 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-admin
2026-06-20 09:18:11,929 [I] DB FLAVOR :POSTGRES
2026-06-20 09:18:11,929 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:18:11,929 [I] Checking connection..
2026-06-20 09:18:11,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 09:18:12,182 [I] Checking connection passed.
2026-06-20 09:18:12,182 [I] --------- Verifying version history table ---------
2026-06-20 09:18:12,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:18:12,436 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:18:12,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:18:12,675 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:18:12,675 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 09:18:12,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 09:18:12.912 UTC [12216] WARNING: there is no transaction in progress
2026-06-20 09:18:12,918 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 09:18:12,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:18:13,173 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 09:18:13,173 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:18:13,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 09:18:13,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 09:18:13,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rq8c.c.gcp-upstream.internal','N') ;"
2026-06-20 09:18:13,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:18:14,156 [I] Table x_portal_user does not exist in database ranger
2026-06-20 09:18:14,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:18:14,384 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 09:18:14,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:18:14,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:18:14,853 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 09:18:14,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 09:18:15.771 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.785 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.800 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.815 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.831 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.848 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.903 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.910 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.925 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.935 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.946 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.960 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.970 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.976 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.982 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:15.988 UTC [12324] WARNING: there is no transaction in progress
2026-06-20 09:18:16,566 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 09:18:16,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 09:18:16,800 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 09:18:16,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:18:17,050 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 09:18:17,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:18:17,307 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 09:18:17,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:18:17,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:18:17,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rq8c.c.gcp-upstream.internal';"
2026-06-20 09:18:18,036 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 09:18:18,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 09:18:18,276 [I] DB_PATCHES have already been applied
I20260620 09:18:18.281482 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7HdOzb/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 09:18:18.281553 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 09:18:18.284425 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:38005
Jun 20, 2026 9:18:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:18:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:18:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:18:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:18:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:18:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:18:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 09:18:19.281719 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.085s user 0.000s sys 0.007s
Jun 20, 2026 9:18:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:18:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 9:18:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:18:38 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:18:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 09:18:39.417866 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 09:18:39 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947119, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 09:18:39.484623 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 09:18:39.492465 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 09:18:40.210373 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 09:18:40.210438 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-kms
I20260620 09:18:40.210453 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53659
2026-06-20 09:18:40,350 [I] DB FLAVOR :POSTGRES
2026-06-20 09:18:40,350 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:18:40,350 [I] Checking connection
2026-06-20 09:18:40,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 09:18:40,567 [I] connection success
2026-06-20 09:18:40,567 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 09:18:40,568 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 09:18:40,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 09:18:40,789 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 09:18:40,789 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:18:40,790 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 09:18:40,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53659/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 09:18:41,030 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 09:18:41.444394 392 mini_ranger_kms.cc:310] Created kms service
I20260620 09:18:41.610711 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 09:18:41.652798 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 09:18:41 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947119, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:42.019052 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 09:18:42.019157 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task7HdOzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7HdOzb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 09:18:42.019176 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 09:18:42.022199 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12504
I20260620 09:18:42.022274 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:59789
Jun 20, 2026 9:18:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:18:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:18:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:18:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:18:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:18:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 09:18:43.984841 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.500s user 0.004s sys 0.119s
I20260620 09:18:43.984921 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 09:18:43.984964 392 mini_ranger_kms.cc:401] 127.0.98.20:59789/kms/v1/keys
Jun 20 09:18:44 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781947124, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 09:18:44 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947124, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:18:44 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947124, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:18:46 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:18:46.994378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33629
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33629
--ranger_config_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59789/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:47.071246 12555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:47.071440 12555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:47.071460 12555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:47.072573 12555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:18:47.072623 12555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:47.072638 12555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:47.072647 12555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:18:47.072659 12555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:18:47.074169 12555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:59789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33629
--ranger_config_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:47.074370 12555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:47.074570 12555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:47.079483 12563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:47.079452 12560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:47.079581 12555 server_base.cc:1034] running on GCE node
W20260620 09:18:47.079640 12561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:47.080036 12555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:47.080359 12555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:47.081532 12555 hybrid_clock.cc:648] HybridClock initialized: now 1781947127081512 us; error 33 us; skew 500 ppm
Jun 20 09:18:47 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:47.084369 12555 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:18:47.084887 12555 webserver.cc:460] Webserver started at http://127.0.98.62:39579/ using document root <none> and password file <none>
I20260620 09:18:47.085106 12555 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:47.085158 12555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:47.085282 12555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:18:47 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:47.139545 12555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "b778b238640e469db403ef57ba597c65"
format_stamp: "Formatted at 2026-06-20 09:18:47 on dist-test-slave-rq8c"
server_key: "87b424d2967c0f2a2feed6e8890854c0"
server_key_iv: "43e9813aaecdb0dc89ea542a3b7b93fa"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:47.139982 12555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "b778b238640e469db403ef57ba597c65"
format_stamp: "Formatted at 2026-06-20 09:18:47 on dist-test-slave-rq8c"
server_key: "87b424d2967c0f2a2feed6e8890854c0"
server_key_iv: "43e9813aaecdb0dc89ea542a3b7b93fa"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:47.141280 12555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:18:47.156560 12570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:47.156754 12555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:18:47.156883 12555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "b778b238640e469db403ef57ba597c65"
format_stamp: "Formatted at 2026-06-20 09:18:47 on dist-test-slave-rq8c"
server_key: "87b424d2967c0f2a2feed6e8890854c0"
server_key_iv: "43e9813aaecdb0dc89ea542a3b7b93fa"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:47.156980 12555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:47.173157 12555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:47.174969 12555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:47.175117 12555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:47.179916 12555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33629
I20260620 09:18:47.179941 12622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33629 every 8 connection(s)
I20260620 09:18:47.180341 12555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:18:47.180995 12623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:47.183152 12623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65: Bootstrap starting.
I20260620 09:18:47.183733 12623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:47.183984 12623 log.cc:824] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:47.184577 12623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65: No bootstrap required, opened a new log
I20260620 09:18:47.186510 12623 raft_consensus.cc:348] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b778b238640e469db403ef57ba597c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } }
I20260620 09:18:47.186664 12623 raft_consensus.cc:374] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:47.186697 12623 raft_consensus.cc:729] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b778b238640e469db403ef57ba597c65, State: Initialized, Role: FOLLOWER
I20260620 09:18:47.186831 12623 consensus_queue.cc:260] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b778b238640e469db403ef57ba597c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } }
I20260620 09:18:47.186904 12623 raft_consensus.cc:388] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:18:47.186934 12623 raft_consensus.cc:482] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:18:47.186978 12623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:47.187465 12623 raft_consensus.cc:504] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b778b238640e469db403ef57ba597c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } }
I20260620 09:18:47.187556 12623 leader_election.cc:302] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b778b238640e469db403ef57ba597c65; no voters:
I20260620 09:18:47.187736 12623 leader_election.cc:288] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:18:47.187783 12626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:18:47.187963 12623 sys_catalog.cc:564] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:18:47.187974 12626 raft_consensus.cc:686] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 1 LEADER]: Becoming Leader. State: Replica: b778b238640e469db403ef57ba597c65, State: Running, Role: LEADER
I20260620 09:18:47.188153 12626 consensus_queue.cc:237] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b778b238640e469db403ef57ba597c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } }
I20260620 09:18:47.188467 12626 sys_catalog.cc:455] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b778b238640e469db403ef57ba597c65. Latest consensus state: current_term: 1 leader_uuid: "b778b238640e469db403ef57ba597c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b778b238640e469db403ef57ba597c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } } }
I20260620 09:18:47.188697 12626 sys_catalog.cc:458] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:47.188639 12628 sys_catalog.cc:455] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b778b238640e469db403ef57ba597c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b778b238640e469db403ef57ba597c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } } }
I20260620 09:18:47.189085 12628 sys_catalog.cc:458] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:47.189368 12623 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 09:18:47.189611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 12555
I20260620 09:18:47.189704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:18:47.189867 392 external_mini_cluster.cc:1428] Setting key ad9e0ef8bc56250005c4fcc2a3227eea
I20260620 09:18:47.191699 12635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:18:47.192600 12635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:18:47.195174 12635 catalog_manager.cc:1347] Generated new cluster ID: b394d5a51a344dfab3267e07139096d0
I20260620 09:18:47.195215 12635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:18:47 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:47.203523 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:47.191521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42906 (local address 127.0.98.62:33629)
0620 09:18:47.191988 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:47.192002 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:47.192050 (+ 48us) server_negotiation.cc:407] Connection header received
0620 09:18:47.192310 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:47.192318 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:47.192375 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:47.193257 (+ 882us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:47.193433 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:47.194901 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:47.195942 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:47.196581 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:47.196985 (+ 404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:47.199346 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:47.199350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:47.199351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:47.199363 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:47.201276 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:47.201673 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:47.201676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:47.201677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:47.201793 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:47.201996 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:47.201997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:47.201997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:47.202323 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:47.202832 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:47.202905 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 09:18:47.203018 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":374,"thread_start_us":200,"threads_started":1}
I20260620 09:18:47.214068 12635 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:18:47.214860 12635 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:18:47.217739 12635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65: Generated new TSK 0
I20260620 09:18:47.217954 12635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:18:47.888764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33629
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:47.991257 12668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:47.991511 12668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:47.991545 12668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:47.993232 12668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:47.993289 12668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:47.993326 12668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:18:47.994910 12668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:59789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:47.995116 12668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:47.995323 12668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:48.000895 12676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:48.000916 12673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:48.001317 12674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:48.001902 12668 server_base.cc:1034] running on GCE node
I20260620 09:18:48.002133 12668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:48.002390 12668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:48.003551 12668 hybrid_clock.cc:648] HybridClock initialized: now 1781947128003540 us; error 30 us; skew 500 ppm
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:48.006611 12668 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:18:48.007112 12668 webserver.cc:460] Webserver started at http://127.0.98.1:42839/ using document root <none> and password file <none>
I20260620 09:18:48.007395 12668 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:48.007517 12668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:48.007722 12668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:48.022244 12668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "eab0735fb0fd4b6d97a2c2201544de9e"
format_stamp: "Formatted at 2026-06-20 09:18:48 on dist-test-slave-rq8c"
server_key: "7d495692a72ddd9a832e4e6adc7effde"
server_key_iv: "9a714e7ef0f69c3a0ef7e8750056b054"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:48.023677 12668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eab0735fb0fd4b6d97a2c2201544de9e"
format_stamp: "Formatted at 2026-06-20 09:18:48 on dist-test-slave-rq8c"
server_key: "7d495692a72ddd9a832e4e6adc7effde"
server_key_iv: "9a714e7ef0f69c3a0ef7e8750056b054"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:48.029141 12668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260620 09:18:48.048094 12685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.048388 12668 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 09:18:48.048537 12668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "eab0735fb0fd4b6d97a2c2201544de9e"
format_stamp: "Formatted at 2026-06-20 09:18:48 on dist-test-slave-rq8c"
server_key: "7d495692a72ddd9a832e4e6adc7effde"
server_key_iv: "9a714e7ef0f69c3a0ef7e8750056b054"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:48.048740 12668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:48.065302 12668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:48.067137 12668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:48.067257 12668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:48.067473 12668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:48.067849 12668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:48.067886 12668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.067915 12668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:48.067952 12668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.074246 12668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34497
I20260620 09:18:48.074282 12798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34497 every 8 connection(s)
I20260620 09:18:48.074697 12668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:48.084362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 12668
I20260620 09:18:48.084462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:18:48.084652 392 external_mini_cluster.cc:1428] Setting key 57637cb88d07f7b0a9046440f654d5f4
I20260620 09:18:48.086598 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:48.076024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33629 (local address 127.0.98.1:43811)
0620 09:18:48.076456 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:48.076502 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:48.077099 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:48.077504 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:48.077509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:48.077732 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:48.077993 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:48.077999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.079012 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.079013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:48.080288 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:48.080291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.080838 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.080839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:48.080970 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:48.081481 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:48.081491 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:48.083048 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:48.085074 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:48.085080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:48.085083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:48.085340 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:48.085551 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:48.085551 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:48.085552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:48.085671 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:48.085934 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:48.085940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:48.086105 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 09:18:48.086209 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 09:18:48.086355 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":84,"threads_started":1}
I20260620 09:18:48.087083 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:48.076294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43811 (local address 127.0.98.62:33629)
0620 09:18:48.077270 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:48.077272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:48.077280 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:18:48.077306 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:48.077306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:48.077318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:48.077376 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:48.078093 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.078908 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.080377 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.080706 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.080766 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:48.083142 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:48.083143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:48.083144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:48.083151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:48.084991 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:48.085417 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:48.085418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:48.085418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:48.085490 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:48.085735 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:48.085736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:48.085736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:48.085859 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:48.085912 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:48.086917 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0620 09:18:48.086993 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":74,"threads_started":1}
I20260620 09:18:48.087786 12799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33629
I20260620 09:18:48.087870 12799 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:48.088037 12799 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
I20260620 09:18:48.088577 12587 ts_manager.cc:194] Registered new tserver with Master: eab0735fb0fd4b6d97a2c2201544de9e (127.0.98.1:34497)
I20260620 09:18:48.089454 12587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43811
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:18:48.115068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33629
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 09:18:48.229943 12807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:48.230353 12807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:48.230428 12807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:48.232239 12807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:48.232345 12807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:48.232422 12807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:18:48.234571 12807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:59789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:48.234943 12807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:48.235219 12807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:48.241289 12815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:48.241528 12812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:48.241621 12807 server_base.cc:1034] running on GCE node
W20260620 09:18:48.241556 12813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:48.241911 12807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:48.242202 12807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:48.243363 12807 hybrid_clock.cc:648] HybridClock initialized: now 1781947128243337 us; error 38 us; skew 500 ppm
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:48.246511 12807 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:18:48.247022 12807 webserver.cc:460] Webserver started at http://127.0.98.2:37743/ using document root <none> and password file <none>
I20260620 09:18:48.247256 12807 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:48.247304 12807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:48.247426 12807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:48.261039 12807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "2fc7a86fb6d648b7806d464f4cb60006"
format_stamp: "Formatted at 2026-06-20 09:18:48 on dist-test-slave-rq8c"
server_key: "93d9ddc36437104f9a4a5b88b20a559b"
server_key_iv: "194f9f8909338ad012778fbb4492628c"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:48.261601 12807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2fc7a86fb6d648b7806d464f4cb60006"
format_stamp: "Formatted at 2026-06-20 09:18:48 on dist-test-slave-rq8c"
server_key: "93d9ddc36437104f9a4a5b88b20a559b"
server_key_iv: "194f9f8909338ad012778fbb4492628c"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:48.263521 12807 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:18:48.279027 12823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.279807 12807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:18:48.279915 12807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "2fc7a86fb6d648b7806d464f4cb60006"
format_stamp: "Formatted at 2026-06-20 09:18:48 on dist-test-slave-rq8c"
server_key: "93d9ddc36437104f9a4a5b88b20a559b"
server_key_iv: "194f9f8909338ad012778fbb4492628c"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:48.280021 12807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:48.308260 12807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:48.310479 12807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:48.310631 12807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:48.310901 12807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:48.311293 12807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:48.311338 12807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.311367 12807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:48.311384 12807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.317727 12807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44359
I20260620 09:18:48.318048 12936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44359 every 8 connection(s)
I20260620 09:18:48.318188 12807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:18:48.321936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 12807
I20260620 09:18:48.322017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:18:48.322198 392 external_mini_cluster.cc:1428] Setting key b9f3f7e94e1d3a65b06071a298207fb1
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:48.329681 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:48.320044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33629 (local address 127.0.98.2:58187)
0620 09:18:48.320192 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:48.320234 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:48.320696 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:48.320946 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:48.320950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:48.321126 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:48.321314 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:48.321318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.322299 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.322300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:48.323390 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:48.323394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.323978 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.323979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:48.324121 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:48.324652 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:48.324663 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:48.326221 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:48.328078 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:48.328084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:48.328086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:48.328368 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:48.328568 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:48.328568 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:48.328569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:48.328711 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:48.329012 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:48.329017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:48.329190 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 09:18:48.329293 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 09:18:48.329445 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260620 09:18:48.330139 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:48.320091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58187 (local address 127.0.98.62:33629)
0620 09:18:48.320155 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:48.320157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:48.320315 (+ 158us) server_negotiation.cc:407] Connection header received
0620 09:18:48.320795 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:48.320796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:48.320808 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:48.320885 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:48.321407 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.322179 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.323489 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.323812 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.323883 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:48.326323 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:48.326325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:48.326325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:48.326333 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:48.327987 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:48.328447 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:48.328448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:48.328449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:48.328507 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:48.328780 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:48.328782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:48.328782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:48.328930 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:48.328987 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:48.329960 (+ 973us) server_negotiation.cc:299] Negotiation successful
0620 09:18:48.330049 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:18:48.330451 12937 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33629
I20260620 09:18:48.330551 12937 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:48.330739 12937 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
I20260620 09:18:48.331154 12587 ts_manager.cc:194] Registered new tserver with Master: 2fc7a86fb6d648b7806d464f4cb60006 (127.0.98.2:44359)
I20260620 09:18:48.331673 12587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58187
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:18:48.355571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33629
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:48.448675 12942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:48.448992 12942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:48.449033 12942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:48.450867 12942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:48.450948 12942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:48.450992 12942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:18:48.453415 12942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:59789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:48.453699 12942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:48.454006 12942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:48.460331 12948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:48.460328 12947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:48.460619 12950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:48.461421 12942 server_base.cc:1034] running on GCE node
I20260620 09:18:48.461644 12942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:48.461927 12942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:48.463122 12942 hybrid_clock.cc:648] HybridClock initialized: now 1781947128463094 us; error 43 us; skew 500 ppm
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:48.466444 12942 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:18:48.467046 12942 webserver.cc:460] Webserver started at http://127.0.98.3:43163/ using document root <none> and password file <none>
I20260620 09:18:48.467388 12942 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:48.467509 12942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:48.467715 12942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:48.491612 12942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "0dee06e39bf14310aed79602b97f35f6"
format_stamp: "Formatted at 2026-06-20 09:18:48 on dist-test-slave-rq8c"
server_key: "f79b215e12f731d69084a74488283b49"
server_key_iv: "1e1a0fa44d281ee69a675795fe4cf0e5"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:48.492120 12942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0dee06e39bf14310aed79602b97f35f6"
format_stamp: "Formatted at 2026-06-20 09:18:48 on dist-test-slave-rq8c"
server_key: "f79b215e12f731d69084a74488283b49"
server_key_iv: "1e1a0fa44d281ee69a675795fe4cf0e5"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:48.494606 12942 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:18:48.519101 12957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.519402 12942 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260620 09:18:48.519510 12942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "0dee06e39bf14310aed79602b97f35f6"
format_stamp: "Formatted at 2026-06-20 09:18:48 on dist-test-slave-rq8c"
server_key: "f79b215e12f731d69084a74488283b49"
server_key_iv: "1e1a0fa44d281ee69a675795fe4cf0e5"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:48.519651 12942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:48.534052 12942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:48.536228 12942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:48.536368 12942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:48.536583 12942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:48.536932 12942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:48.536986 12942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.537020 12942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:48.537042 12942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.544970 12942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41981
I20260620 09:18:48.545434 12942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:18:48.551832 13070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41981 every 8 connection(s)
I20260620 09:18:48.554490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 12942
I20260620 09:18:48.554584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:18:48.554740 392 external_mini_cluster.cc:1428] Setting key ddb10b7438dd1bfcbaae8d6ea2021163
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:48.562461 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:48.546649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37179 (local address 127.0.98.62:33629)
0620 09:18:48.546693 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:48.546695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:48.552229 (+ 5534us) server_negotiation.cc:407] Connection header received
0620 09:18:48.552972 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:48.552973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:48.552986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:48.553057 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:48.553867 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.554669 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.556600 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.556898 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.556957 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:48.559376 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:48.559377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:48.559377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:48.559385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:48.560945 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:48.561383 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:48.561384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:48.561385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:48.561447 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:48.561733 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:48.561734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:48.561734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:48.561871 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:48.561952 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:48.562288 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 09:18:48.562364 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:18:48.562932 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:48.546358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33629 (local address 127.0.98.3:37179)
0620 09:18:48.552050 (+ 5692us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:48.552106 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:48.552844 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:48.553193 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:48.553197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:48.553497 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:48.553767 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:48.553774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.554778 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.554779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:48.556499 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:48.556502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.557063 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.557063 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:48.557196 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:48.557771 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:48.557781 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:48.559285 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:48.561031 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:48.561039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:48.561041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:48.561307 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:48.561510 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:48.561511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:48.561511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:48.561684 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:48.561937 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:48.561942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:48.562141 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 09:18:48.562573 (+ 432us) client_negotiation.cc:240] Negotiation successful
0620 09:18:48.562720 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5638,"thread_start_us":87,"threads_started":1}
I20260620 09:18:48.563345 13071 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33629
I20260620 09:18:48.563444 13071 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:48.563651 13071 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
I20260620 09:18:48.564043 12587 ts_manager.cc:194] Registered new tserver with Master: 0dee06e39bf14310aed79602b97f35f6 (127.0.98.3:41981)
I20260620 09:18:48.564522 12587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37179
I20260620 09:18:48.566293 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:18:48.567011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 12668
I20260620 09:18:48.572894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 12807
I20260620 09:18:48.577411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 12942
I20260620 09:18:48.582647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 12555
I20260620 09:18:48.587396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33629
--webserver_interface=127.0.98.62
--webserver_port=39579
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33629
--ranger_config_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59789/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59789/kms with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:48.664813 13074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:48.665024 13074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:48.665045 13074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:48.666220 13074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:18:48.666262 13074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:48.666275 13074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:48.666289 13074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:18:48.666312 13074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:18:48.667913 13074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33629
--ranger_config_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:48.668128 13074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:48.668332 13074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:48.672879 13082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:48.672855 13080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:48.672861 13079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:48.673113 13074 server_base.cc:1034] running on GCE node
I20260620 09:18:48.673309 13074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:48.673556 13074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:48.674721 13074 hybrid_clock.cc:648] HybridClock initialized: now 1781947128674709 us; error 27 us; skew 500 ppm
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:48.677126 13074 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:18:48.677560 13074 webserver.cc:460] Webserver started at http://127.0.98.62:39579/ using document root <none> and password file <none>
I20260620 09:18:48.677747 13074 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:48.677796 13074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:48.750250 13074 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:18:48.750979 13090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.751138 13074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:18:48.751230 13074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "b778b238640e469db403ef57ba597c65"
format_stamp: "Formatted at 2026-06-20 09:18:47 on dist-test-slave-rq8c"
server_key: "87b424d2967c0f2a2feed6e8890854c0"
server_key_iv: "43e9813aaecdb0dc89ea542a3b7b93fa"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:48.751601 13074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:48.781616 13074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:48.783457 13074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:48.783612 13074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:48.787820 13074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33629
I20260620 09:18:48.787889 13142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33629 every 8 connection(s)
I20260620 09:18:48.788234 13074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:18:48.789431 13143 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:18:48.790079 13143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65: Bootstrap starting.
I20260620 09:18:48.792833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 13074
I20260620 09:18:48.793104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34497
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33629
--webserver_port=42839
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59789/kms with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:18:48.798105 13143 log.cc:824] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:48.799865 13143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:18:48.800110 13143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65: Bootstrap complete.
I20260620 09:18:48.802280 13143 raft_consensus.cc:348] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b778b238640e469db403ef57ba597c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } }
I20260620 09:18:48.802520 13143 raft_consensus.cc:729] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b778b238640e469db403ef57ba597c65, State: Initialized, Role: FOLLOWER
I20260620 09:18:48.802677 13143 consensus_queue.cc:260] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b778b238640e469db403ef57ba597c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } }
I20260620 09:18:48.802747 13143 raft_consensus.cc:388] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:18:48.802791 13143 raft_consensus.cc:482] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:18:48.802837 13143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:18:48.803840 13143 raft_consensus.cc:504] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b778b238640e469db403ef57ba597c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } }
I20260620 09:18:48.803932 13143 leader_election.cc:302] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b778b238640e469db403ef57ba597c65; no voters:
I20260620 09:18:48.804083 13143 leader_election.cc:288] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:18:48.804155 13147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:18:48.804322 13143 sys_catalog.cc:564] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:18:48.804337 13147 raft_consensus.cc:686] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [term 2 LEADER]: Becoming Leader. State: Replica: b778b238640e469db403ef57ba597c65, State: Running, Role: LEADER
I20260620 09:18:48.804461 13147 consensus_queue.cc:237] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b778b238640e469db403ef57ba597c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } }
I20260620 09:18:48.804817 13148 sys_catalog.cc:455] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b778b238640e469db403ef57ba597c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b778b238640e469db403ef57ba597c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } } }
I20260620 09:18:48.804831 13149 sys_catalog.cc:455] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b778b238640e469db403ef57ba597c65. Latest consensus state: current_term: 2 leader_uuid: "b778b238640e469db403ef57ba597c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b778b238640e469db403ef57ba597c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } } }
I20260620 09:18:48.805037 13149 sys_catalog.cc:458] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:48.805198 13148 sys_catalog.cc:458] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:48.805311 13157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:18:48.805765 13157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:18:48.805883 13157 catalog_manager.cc:1259] Loaded cluster ID: b394d5a51a344dfab3267e07139096d0
I20260620 09:18:48.805917 13157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:18:48.806183 13143 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 09:18:48.807117 13157 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:18:48.807461 13157 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65: Loaded TSK: 0
I20260620 09:18:48.807709 13157 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:18:48.894558 13145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:48.894757 13145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:48.894789 13145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:48.895998 13145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:48.896037 13145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:48.896075 13145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:18:48.897806 13145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42839
--tserver_master_addrs=127.0.98.62:33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:48.898072 13145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:48.898329 13145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:48.903448 13180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:48.903448 13181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:48.903477 13183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:48.904181 13145 server_base.cc:1034] running on GCE node
I20260620 09:18:48.904412 13145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:48.904682 13145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:48.905871 13145 hybrid_clock.cc:648] HybridClock initialized: now 1781947128905851 us; error 30 us; skew 500 ppm
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:48.908388 13145 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:18:48.908828 13145 webserver.cc:460] Webserver started at http://127.0.98.1:42839/ using document root <none> and password file <none>
I20260620 09:18:48.909045 13145 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:48.909106 13145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:48.940158 13145 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:18:48.940896 13191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.941042 13145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:18:48.941124 13145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "eab0735fb0fd4b6d97a2c2201544de9e"
format_stamp: "Formatted at 2026-06-20 09:18:48 on dist-test-slave-rq8c"
server_key: "7d495692a72ddd9a832e4e6adc7effde"
server_key_iv: "9a714e7ef0f69c3a0ef7e8750056b054"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:48.941568 13145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:48.950680 13145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:48.953001 13145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:48.953140 13145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:48.953351 13145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:48.953672 13145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:48.953706 13145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.953737 13145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:48.953755 13145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:48.960166 13145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34497
I20260620 09:18:48.960310 13304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34497 every 8 connection(s)
I20260620 09:18:48.960566 13145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:18:48.968273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 13145
I20260620 09:18:48.968561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44359
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33629
--webserver_port=37743
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59789/kms with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:18:48 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:48.975135 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:48.961328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47591 (local address 127.0.98.62:33629)
0620 09:18:48.962214 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:48.962222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:48.962259 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:18:48.962398 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:48.962405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:48.962457 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:48.962592 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:48.963348 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.964784 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.966227 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.966724 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.966838 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:48.970950 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:48.970955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:48.970956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:48.970969 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:48.973021 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:48.973556 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:48.973560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:48.973560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:48.973673 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:48.973976 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:48.973977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:48.973978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:48.974291 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:48.974354 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:48.974795 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 09:18:48.974927 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":102,"threads_started":1}
I20260620 09:18:48.975734 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:48.961241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33629 (local address 127.0.98.1:47591)
0620 09:18:48.961582 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:48.961618 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:48.962056 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:48.962665 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:48.962670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:48.962919 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:48.963208 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:48.963216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.964900 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.964901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:48.966138 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:48.966141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:48.966838 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:48.966839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:48.966981 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:48.967600 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:48.967611 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:48.970807 (+ 3196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:48.973131 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:48.973138 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:48.973141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:48.973452 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:48.973754 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:48.973755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:48.973755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:48.973895 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:48.974417 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:48.974424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:48.974596 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 09:18:48.975311 (+ 715us) client_negotiation.cc:240] Negotiation successful
0620 09:18:48.975492 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":73,"threads_started":1}
I20260620 09:18:48.976451 13305 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33629
I20260620 09:18:48.976547 13305 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:48.976716 13305 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
W20260620 09:18:48.977591 13305 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33629 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 09:18:48.977670 13305 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:48.977741 13305 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
I20260620 09:18:48.978106 13305 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:48.978186 13305 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
W20260620 09:18:48.978487 13305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:18:49.063432 13309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:49.063647 13309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:49.063683 13309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:49.065353 13309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:49.065428 13309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:49.065471 13309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:18:49.067588 13309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37743
--tserver_master_addrs=127.0.98.62:33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:49.067852 13309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:49.068101 13309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:49.074625 13317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:49.074633 13314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:49.074633 13315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:49.075080 13309 server_base.cc:1034] running on GCE node
I20260620 09:18:49.075287 13309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:49.075578 13309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:49.076769 13309 hybrid_clock.cc:648] HybridClock initialized: now 1781947129076749 us; error 32 us; skew 500 ppm
Jun 20 09:18:49 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:49.079754 13309 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:18:49.080250 13309 webserver.cc:460] Webserver started at http://127.0.98.2:37743/ using document root <none> and password file <none>
I20260620 09:18:49.080492 13309 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:49.080554 13309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:18:49 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:49.121047 13309 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260620 09:18:49.121757 13325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:49.121914 13309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:18:49.122014 13309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "2fc7a86fb6d648b7806d464f4cb60006"
format_stamp: "Formatted at 2026-06-20 09:18:48 on dist-test-slave-rq8c"
server_key: "93d9ddc36437104f9a4a5b88b20a559b"
server_key_iv: "194f9f8909338ad012778fbb4492628c"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:49.122406 13309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:49.131459 13309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:49.133524 13309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:49.133664 13309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:49.133898 13309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:49.134251 13309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:49.134298 13309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:49.134330 13309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:49.134346 13309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:49.140765 13309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44359
I20260620 09:18:49.141251 13309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:18:49.141899 13438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44359 every 8 connection(s)
I20260620 09:18:49.144557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 13309
I20260620 09:18:49.144841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41981
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33629
--webserver_port=43163
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59789/kms with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:18:49 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:49.153563 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:49.142337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60307 (local address 127.0.98.62:33629)
0620 09:18:49.142390 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:49.142392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:49.142403 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:18:49.143044 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:49.143046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:49.143061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:49.143128 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:49.143876 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:49.144603 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:49.147472 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:49.147774 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:49.147839 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:49.150168 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:49.150170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:49.150170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:49.150178 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:49.151872 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:49.152389 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:49.152391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:49.152391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:49.152454 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:49.152751 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:49.152752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:49.152752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:49.152899 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:49.152945 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:49.153359 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 09:18:49.153463 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:18:49.154414 13441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:49.142019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33629 (local address 127.0.98.2:60307)
0620 09:18:49.142200 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:49.142247 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:49.142918 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:49.143204 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:49.143209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:49.143453 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:49.143767 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:49.143774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:49.146241 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:49.146243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:49.147376 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:49.147380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:49.147915 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:49.147916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:49.148050 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:49.148650 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:49.148659 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:49.150052 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:49.151966 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:49.151974 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:49.151977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:49.152284 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:49.152529 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:49.152530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:49.152530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:49.152675 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:49.152993 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:49.152999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:49.153197 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 09:18:49.153968 (+ 771us) client_negotiation.cc:240] Negotiation successful
0620 09:18:49.154153 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260620 09:18:49.155145 13439 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33629
I20260620 09:18:49.155339 13439 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:49.155661 13439 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
W20260620 09:18:49.156391 13439 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33629 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 09:18:49.156507 13439 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:49.156605 13439 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
I20260620 09:18:49.156916 13439 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:49.157019 13439 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
W20260620 09:18:49.157289 13439 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:18:49.251641 13442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:49.251920 13442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:49.251955 13442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:49.253692 13442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:49.253752 13442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:49.253793 13442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:18:49.256037 13442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43163
--tserver_master_addrs=127.0.98.62:33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:49.256292 13442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:49.256577 13442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:49.262852 13450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:49.262887 13448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:49.264019 13447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:49.264638 13442 server_base.cc:1034] running on GCE node
I20260620 09:18:49.264845 13442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:49.265125 13442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:49.266295 13442 hybrid_clock.cc:648] HybridClock initialized: now 1781947129266167 us; error 139 us; skew 500 ppm
Jun 20 09:18:49 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:49.269582 13442 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:18:49.270135 13442 webserver.cc:460] Webserver started at http://127.0.98.3:43163/ using document root <none> and password file <none>
I20260620 09:18:49.270403 13442 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:49.270478 13442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:18:49 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:49.299230 13442 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:18:49.300155 13459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:49.300351 13442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:18:49.300443 13442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "0dee06e39bf14310aed79602b97f35f6"
format_stamp: "Formatted at 2026-06-20 09:18:48 on dist-test-slave-rq8c"
server_key: "f79b215e12f731d69084a74488283b49"
server_key_iv: "1e1a0fa44d281ee69a675795fe4cf0e5"
server_key_version: "kuduclusterkey@0"
I20260620 09:18:49.300884 13442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:49.327366 13442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:49.329365 13442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:49.329625 13442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:49.329923 13442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:49.330320 13442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:49.330372 13442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:49.330405 13442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:49.330430 13442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:49.337069 13442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41981
I20260620 09:18:49.337144 13572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41981 every 8 connection(s)
I20260620 09:18:49.337478 13442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:18:49 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:49.344833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 13442
I20260620 09:18:49.347906 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:49.338191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58629 (local address 127.0.98.62:33629)
0620 09:18:49.338781 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:49.338782 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:49.338793 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:18:49.339173 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:49.339175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:49.339194 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:49.339264 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:49.339966 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:49.340460 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:49.341896 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:49.342073 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:49.342118 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:49.344450 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:49.344451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:49.344452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:49.344460 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:49.346268 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:49.346762 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:49.346763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:49.346763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:49.346826 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:49.347120 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:49.347121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:49.347122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:49.347259 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:49.347324 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:49.347708 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 09:18:49.347801 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558}
I20260620 09:18:49.348075 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:49.338197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33629 (local address 127.0.98.3:58629)
0620 09:18:49.338357 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:49.338403 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:49.339032 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:49.339351 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:49.339355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:49.339580 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:49.339879 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:49.339884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:49.340538 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:49.340539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:49.341821 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:49.341824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:49.342160 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:49.342162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:49.342302 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:49.342871 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:49.342881 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:49.344352 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:49.346360 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:49.346367 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:49.346370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:49.346682 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:49.346896 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:49.346897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:49.346897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:49.347051 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:49.347326 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:49.347332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:49.347514 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 09:18:49.347666 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 09:18:49.347830 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260620 09:18:49.348762 13573 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33629
I20260620 09:18:49.348866 13573 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:49.349058 13573 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
W20260620 09:18:49.349689 13573 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33629 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 09:18:49.349766 13573 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:49.349833 13573 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
I20260620 09:18:49.350147 13573 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:49.350224 13573 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
W20260620 09:18:49.350507 13573 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:18:49.359987 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:49.345445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.0.98.62:33629)
0620 09:18:49.345579 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:49.345581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:49.348566 (+ 2985us) server_negotiation.cc:407] Connection header received
0620 09:18:49.348632 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:49.348633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:49.348645 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:49.348711 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:49.353898 (+ 5187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:49.354460 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:49.355664 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:49.355898 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:49.355961 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:49.357154 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:49.357155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:49.357156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:49.357163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:49.358769 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:49.359097 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:49.359098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:49.359098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:49.359173 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:49.359390 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:49.359391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:49.359391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:49.359529 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:49.359741 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:49.359783 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 09:18:49.359881 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
Jun 20 09:18:49 dist-test-slave-rq8c krb5kdc[12111](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781947129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:49.978677 13305 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:49.978832 13305 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
I20260620 09:18:49.979247 13107 ts_manager.cc:194] Registered new tserver with Master: eab0735fb0fd4b6d97a2c2201544de9e (127.0.98.1:34497)
I20260620 09:18:49.980307 13107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47591
I20260620 09:18:50.157487 13439 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:50.157653 13439 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
I20260620 09:18:50.157970 13106 ts_manager.cc:194] Registered new tserver with Master: 2fc7a86fb6d648b7806d464f4cb60006 (127.0.98.2:44359)
I20260620 09:18:50.158512 13106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60307
Jun 20 09:18:50 dist-test-slave-rq8c krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:18:50.350674 13573 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:50.350813 13573 heartbeater.cc:511] Master 127.0.98.62:33629 requested a full tablet report, sending...
I20260620 09:18:50.351090 13106 ts_manager.cc:194] Registered new tserver with Master: 0dee06e39bf14310aed79602b97f35f6 (127.0.98.3:41981)
I20260620 09:18:50.351478 13106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58629
I20260620 09:18:50.360364 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:18:50.367378 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.361438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.0.98.62:33629)
0620 09:18:50.361551 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:50.361553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:50.361562 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:18:50.361638 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:50.361638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:50.361649 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:50.361705 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:50.362227 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.362731 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.363434 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.363677 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.363743 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:50.364919 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:50.364921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:50.364921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:50.364929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:50.366285 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:50.366595 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:50.366596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:50.366596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:50.366650 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:50.366813 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:50.366814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:50.366814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:50.366929 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:50.367014 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:50.367118 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 09:18:50.367257 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
I20260620 09:18:50.369355 13106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:18:50.371008 13106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:18:50.378188 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.373590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34497 (local address 127.0.0.1:41970)
0620 09:18:50.374115 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:50.374164 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:50.374232 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:50.374540 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:50.374543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:50.374545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:50.374621 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:50.374624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.375441 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.375442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:50.376904 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:50.376907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.377910 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.377911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:50.377993 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:50.378006 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 09:18:50.378062 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 09:18:50.378108 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":91,"threads_started":1}
I20260620 09:18:50.378619 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.373871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.0.98.1:34497)
0620 09:18:50.374050 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:50.374056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:50.374361 (+ 305us) server_negotiation.cc:407] Connection header received
0620 09:18:50.374400 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:50.374402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:50.374419 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:50.374538 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:50.374683 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.375367 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.377000 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.377736 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.377784 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:50.377853 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:50.378506 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 09:18:50.378549 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":57,"threads_started":1}
I20260620 09:18:50.379541 13239 tablet_service.cc:1467] Processing CreateTablet for tablet 1969dee8268d4a6290e9cbe16b6dd5a7 (DEFAULT_TABLE table=test-table [id=42a54d218d2842df8998c3502db81657]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:18:50.379869 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.373590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41981 (local address 127.0.0.1:34704)
0620 09:18:50.375784 (+ 2194us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:50.375794 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:50.375845 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:50.376096 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:50.376097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:50.376098 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:50.376194 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:50.376196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.377390 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.377391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:50.378688 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:50.378691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.379658 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.379659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:50.379722 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:50.379726 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:18:50.379772 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 09:18:50.379808 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2176,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260620 09:18:50.379930 13239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1969dee8268d4a6290e9cbe16b6dd5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:50.380126 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.374005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.0.98.3:41981)
0620 09:18:50.374131 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:50.374134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:50.375904 (+ 1770us) server_negotiation.cc:407] Connection header received
0620 09:18:50.375944 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:50.375947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:50.375961 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:50.376026 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:50.376267 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.377309 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.378765 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.379457 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.379522 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:50.379589 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:50.380007 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 09:18:50.380053 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":49,"threads_started":1}
I20260620 09:18:50.380751 13507 tablet_service.cc:1467] Processing CreateTablet for tablet 1969dee8268d4a6290e9cbe16b6dd5a7 (DEFAULT_TABLE table=test-table [id=42a54d218d2842df8998c3502db81657]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:18:50.381083 13507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1969dee8268d4a6290e9cbe16b6dd5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:50.384359 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.373816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44359 (local address 127.0.0.1:44882)
0620 09:18:50.380392 (+ 6576us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:50.380403 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:50.380454 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:50.380712 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:50.380713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:50.380715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:50.380749 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:50.380751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.382237 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.382238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:50.383266 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:50.383269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.384128 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.384129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:50.384200 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:50.384205 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:18:50.384255 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 09:18:50.384292 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6560,"thread_start_us":68,"threads_started":1}
I20260620 09:18:50.384572 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.373951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.0.98.2:44359)
0620 09:18:50.375728 (+ 1777us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:50.375732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:50.380511 (+ 4779us) server_negotiation.cc:407] Connection header received
0620 09:18:50.380552 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:50.380555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:50.380571 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:50.380636 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:50.380817 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.382125 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.383373 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.384047 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.384115 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:50.384186 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:50.384460 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 09:18:50.384499 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1741,"thread_start_us":63,"threads_started":1}
I20260620 09:18:50.385232 13370 tablet_service.cc:1467] Processing CreateTablet for tablet 1969dee8268d4a6290e9cbe16b6dd5a7 (DEFAULT_TABLE table=test-table [id=42a54d218d2842df8998c3502db81657]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:18:50.385505 13370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1969dee8268d4a6290e9cbe16b6dd5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:50.387785 13607 tablet_bootstrap.cc:492] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6: Bootstrap starting.
I20260620 09:18:50.388331 13607 tablet_bootstrap.cc:654] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:50.388603 13607 log.cc:824] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:50.391773 13606 tablet_bootstrap.cc:492] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e: Bootstrap starting.
I20260620 09:18:50.392313 13606 tablet_bootstrap.cc:654] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:50.392565 13606 log.cc:824] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:50.392894 13608 tablet_bootstrap.cc:492] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006: Bootstrap starting.
I20260620 09:18:50.393265 13606 tablet_bootstrap.cc:492] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e: No bootstrap required, opened a new log
I20260620 09:18:50.393343 13606 ts_tablet_manager.cc:1397] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:18:50.395694 13606 raft_consensus.cc:348] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dee06e39bf14310aed79602b97f35f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41981 } } peers { permanent_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34497 } } peers { permanent_uuid: "2fc7a86fb6d648b7806d464f4cb60006" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44359 } }
I20260620 09:18:50.395834 13606 raft_consensus.cc:374] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:50.395865 13606 raft_consensus.cc:729] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab0735fb0fd4b6d97a2c2201544de9e, State: Initialized, Role: FOLLOWER
I20260620 09:18:50.396018 13606 consensus_queue.cc:260] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dee06e39bf14310aed79602b97f35f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41981 } } peers { permanent_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34497 } } peers { permanent_uuid: "2fc7a86fb6d648b7806d464f4cb60006" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44359 } }
I20260620 09:18:50.396255 13606 ts_tablet_manager.cc:1428] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:18:50.396381 13305 heartbeater.cc:503] Master 127.0.98.62:33629 was elected leader, sending a full tablet report...
I20260620 09:18:50.397308 13608 tablet_bootstrap.cc:654] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:50.397554 13608 log.cc:824] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:50.398233 13608 tablet_bootstrap.cc:492] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006: No bootstrap required, opened a new log
I20260620 09:18:50.398303 13608 ts_tablet_manager.cc:1397] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 09:18:50.400154 13608 raft_consensus.cc:348] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dee06e39bf14310aed79602b97f35f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41981 } } peers { permanent_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34497 } } peers { permanent_uuid: "2fc7a86fb6d648b7806d464f4cb60006" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44359 } }
I20260620 09:18:50.400705 13607 tablet_bootstrap.cc:492] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6: No bootstrap required, opened a new log
I20260620 09:18:50.400772 13607 ts_tablet_manager.cc:1397] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260620 09:18:50.402524 13607 raft_consensus.cc:348] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dee06e39bf14310aed79602b97f35f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41981 } } peers { permanent_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34497 } } peers { permanent_uuid: "2fc7a86fb6d648b7806d464f4cb60006" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44359 } }
I20260620 09:18:50.402634 13607 raft_consensus.cc:374] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:50.402653 13607 raft_consensus.cc:729] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dee06e39bf14310aed79602b97f35f6, State: Initialized, Role: FOLLOWER
I20260620 09:18:50.402745 13607 consensus_queue.cc:260] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dee06e39bf14310aed79602b97f35f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41981 } } peers { permanent_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34497 } } peers { permanent_uuid: "2fc7a86fb6d648b7806d464f4cb60006" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44359 } }
I20260620 09:18:50.402920 13608 raft_consensus.cc:374] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:50.402987 13607 ts_tablet_manager.cc:1428] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:18:50.403010 13608 raft_consensus.cc:729] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc7a86fb6d648b7806d464f4cb60006, State: Initialized, Role: FOLLOWER
I20260620 09:18:50.403179 13573 heartbeater.cc:503] Master 127.0.98.62:33629 was elected leader, sending a full tablet report...
I20260620 09:18:50.403133 13608 consensus_queue.cc:260] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dee06e39bf14310aed79602b97f35f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41981 } } peers { permanent_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34497 } } peers { permanent_uuid: "2fc7a86fb6d648b7806d464f4cb60006" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44359 } }
I20260620 09:18:50.403375 13608 ts_tablet_manager.cc:1428] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260620 09:18:50.403471 13439 heartbeater.cc:503] Master 127.0.98.62:33629 was elected leader, sending a full tablet report...
I20260620 09:18:50.432286 13611 raft_consensus.cc:482] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:18:50.432430 13611 raft_consensus.cc:504] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dee06e39bf14310aed79602b97f35f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41981 } } peers { permanent_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34497 } } peers { permanent_uuid: "2fc7a86fb6d648b7806d464f4cb60006" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44359 } }
I20260620 09:18:50.432817 13611 leader_election.cc:288] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dee06e39bf14310aed79602b97f35f6 (127.0.98.3:41981), 2fc7a86fb6d648b7806d464f4cb60006 (127.0.98.2:44359)
I20260620 09:18:50.436833 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.433343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42201 (local address 127.0.98.2:44359)
0620 09:18:50.433382 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:50.433384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:50.433475 (+ 91us) server_negotiation.cc:407] Connection header received
0620 09:18:50.433560 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:50.433561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:50.433573 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:50.433598 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:50.433767 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.434442 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.435828 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.436402 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.436450 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:50.436461 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:50.436719 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 09:18:50.436761 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:18:50.437234 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.433184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44359 (local address 127.0.98.1:42201)
0620 09:18:50.433427 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:50.433435 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:50.433494 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:50.433653 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:50.433653 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:50.433655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:50.433702 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:50.433703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.434520 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.434521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:50.435729 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:50.435733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.436538 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.436539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:50.436606 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:50.436610 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:18:50.437103 (+ 493us) client_negotiation.cc:240] Negotiation successful
0620 09:18:50.437146 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":34,"thread_start_us":59,"threads_started":1}
I20260620 09:18:50.437698 13393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1969dee8268d4a6290e9cbe16b6dd5a7" candidate_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc7a86fb6d648b7806d464f4cb60006" is_pre_election: true
I20260620 09:18:50.437870 13393 raft_consensus.cc:2454] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eab0735fb0fd4b6d97a2c2201544de9e in term 0.
I20260620 09:18:50.438094 13194 leader_election.cc:302] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc7a86fb6d648b7806d464f4cb60006, eab0735fb0fd4b6d97a2c2201544de9e; no voters:
I20260620 09:18:50.438217 13611 raft_consensus.cc:2781] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:18:50.438257 13611 raft_consensus.cc:482] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:18:50.438279 13611 raft_consensus.cc:3037] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:50.439136 13611 raft_consensus.cc:504] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dee06e39bf14310aed79602b97f35f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41981 } } peers { permanent_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34497 } } peers { permanent_uuid: "2fc7a86fb6d648b7806d464f4cb60006" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44359 } }
I20260620 09:18:50.439204 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.433025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57547 (local address 127.0.98.3:41981)
0620 09:18:50.433085 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:50.433087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:50.433204 (+ 117us) server_negotiation.cc:407] Connection header received
0620 09:18:50.433310 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:50.433311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:50.433324 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:50.433352 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:50.433595 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.434365 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.438320 (+ 3955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.438848 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.438893 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:50.438903 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:50.439098 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 09:18:50.439145 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:18:50.439249 13611 leader_election.cc:288] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [CANDIDATE]: Term 1 election: Requested vote from peers 0dee06e39bf14310aed79602b97f35f6 (127.0.98.3:41981), 2fc7a86fb6d648b7806d464f4cb60006 (127.0.98.2:44359)
I20260620 09:18:50.439411 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.432973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41981 (local address 127.0.98.1:57547)
0620 09:18:50.433147 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:50.433157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:50.433233 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:50.433421 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:50.433422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:50.433424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:50.433525 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:50.433526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.434456 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.434457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:50.438206 (+ 3749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:50.438209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.438957 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.438958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:50.439021 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:50.439025 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:18:50.439280 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 09:18:50.439322 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":31,"thread_start_us":103,"threads_started":1}
I20260620 09:18:50.439680 13393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1969dee8268d4a6290e9cbe16b6dd5a7" candidate_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc7a86fb6d648b7806d464f4cb60006"
I20260620 09:18:50.439772 13393 raft_consensus.cc:3037] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:50.439795 13526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1969dee8268d4a6290e9cbe16b6dd5a7" candidate_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dee06e39bf14310aed79602b97f35f6"
I20260620 09:18:50.439898 13526 raft_consensus.cc:3037] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:50.440583 13393 raft_consensus.cc:2454] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eab0735fb0fd4b6d97a2c2201544de9e in term 1.
I20260620 09:18:50.440659 13526 raft_consensus.cc:2454] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eab0735fb0fd4b6d97a2c2201544de9e in term 1.
I20260620 09:18:50.439806 13527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1969dee8268d4a6290e9cbe16b6dd5a7" candidate_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dee06e39bf14310aed79602b97f35f6" is_pre_election: true
I20260620 09:18:50.440827 13527 raft_consensus.cc:2362] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate eab0735fb0fd4b6d97a2c2201544de9e in term 1. Re-sending same reply.
I20260620 09:18:50.440834 13194 leader_election.cc:302] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc7a86fb6d648b7806d464f4cb60006, eab0735fb0fd4b6d97a2c2201544de9e; no voters:
I20260620 09:18:50.440966 13611 raft_consensus.cc:2781] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:18:50.441119 13611 raft_consensus.cc:686] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [term 1 LEADER]: Becoming Leader. State: Replica: eab0735fb0fd4b6d97a2c2201544de9e, State: Running, Role: LEADER
I20260620 09:18:50.441205 13611 consensus_queue.cc:237] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dee06e39bf14310aed79602b97f35f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41981 } } peers { permanent_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34497 } } peers { permanent_uuid: "2fc7a86fb6d648b7806d464f4cb60006" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44359 } }
I20260620 09:18:50.442021 13105 catalog_manager.cc:5495] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e reported cstate change: term changed from 0 to 1, leader changed from <none> to eab0735fb0fd4b6d97a2c2201544de9e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dee06e39bf14310aed79602b97f35f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eab0735fb0fd4b6d97a2c2201544de9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fc7a86fb6d648b7806d464f4cb60006" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44359 } health_report { overall_health: UNKNOWN } } }
I20260620 09:18:50.450395 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.447403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.0.98.1:34497)
0620 09:18:50.447470 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:50.447472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:50.447482 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:18:50.447519 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:50.447520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:50.447537 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:50.447640 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:18:50.447779 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.448442 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.449521 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:50.449806 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:50.449889 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:50.450047 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:18:50.450173 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:18:50.450253 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:50.450297 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 09:18:50.450320 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:18:50.452349 13527 raft_consensus.cc:1261] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6 [term 1 FOLLOWER]: Refusing update from remote peer eab0735fb0fd4b6d97a2c2201544de9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:18:50.452349 13393 raft_consensus.cc:1261] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 2fc7a86fb6d648b7806d464f4cb60006 [term 1 FOLLOWER]: Refusing update from remote peer eab0735fb0fd4b6d97a2c2201544de9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:18:50.452692 13611 consensus_queue.cc:1035] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc7a86fb6d648b7806d464f4cb60006" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:18:50.452898 13617 consensus_queue.cc:1035] T 1969dee8268d4a6290e9cbe16b6dd5a7 P eab0735fb0fd4b6d97a2c2201544de9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dee06e39bf14310aed79602b97f35f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:18:50.457698 13621 mvcc.cc:205] Tried to move back new op lower bound from 7298855446330249216 to 7298855446287736832. Current Snapshot: MvccSnapshot[applied={T|T < 7298855446330249216}]
I20260620 09:18:50.457894 13620 mvcc.cc:205] Tried to move back new op lower bound from 7298855446330249216 to 7298855446287736832. Current Snapshot: MvccSnapshot[applied={T|T < 7298855446330249216}]
I20260620 09:18:50.462037 13105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:18:50.462129 13105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:18:50.462955 13105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b778b238640e469db403ef57ba597c65: Sending DeleteTablet for 3 replicas of tablet 1969dee8268d4a6290e9cbe16b6dd5a7
I20260620 09:18:50.463263 13239 tablet_service.cc:1514] Processing DeleteTablet for tablet 1969dee8268d4a6290e9cbe16b6dd5a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41970
I20260620 09:18:50.463752 13507 tablet_service.cc:1514] Processing DeleteTablet for tablet 1969dee8268d4a6290e9cbe16b6dd5a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34704
I20260620 09:18:50.463990 13370 tablet_service.cc:1514] Processing DeleteTablet for tablet 1969dee8268d4a6290e9cbe16b6dd5a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44882
I20260620 09:18:50.464632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 13145
I20260620 09:18:50.465603 13631 tablet_replica.cc:330] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6: stopping tablet replica
I20260620 09:18:50.465706 13631 raft_consensus.cc:2229] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:18:50.465806 13631 raft_consensus.cc:2258] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:18:50.466240 13631 ts_tablet_manager.cc:1905] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:18:50.469605 13631 ts_tablet_manager.cc:1918] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:18:50.469687 13631 log.cc:1196] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/1969dee8268d4a6290e9cbe16b6dd5a7
I20260620 09:18:50.469959 13631 ts_tablet_manager.cc:1939] T 1969dee8268d4a6290e9cbe16b6dd5a7 P 0dee06e39bf14310aed79602b97f35f6: Deleting consensus metadata
I20260620 09:18:50.470382 13093 catalog_manager.cc:4841] TS 0dee06e39bf14310aed79602b97f35f6 (127.0.98.3:41981): tablet 1969dee8268d4a6290e9cbe16b6dd5a7 (table test-table [id=42a54d218d2842df8998c3502db81657]) successfully deleted
W20260620 09:18:50.471439 13092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34497 (error 108)
I20260620 09:18:50.472285 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.472175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34497 (local address 127.0.0.1:41988)
0620 09:18:50.472216 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:50.472253 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 09:18:50.472333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 13309
W20260620 09:18:50.474762 13092 catalog_manager.cc:4584] TS eab0735fb0fd4b6d97a2c2201544de9e (127.0.98.1:34497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1969dee8268d4a6290e9cbe16b6dd5a7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34497: connect: Connection refused (error 111)
I20260620 09:18:50.478084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 13442
I20260620 09:18:50.478645 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:50.478540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44359 (local address 127.0.0.1:44896)
0620 09:18:50.478588 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:50.478612 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 09:18:50.483094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 13074
I20260620 09:18:50.487887 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 09:18:50.821936 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 09:18:50.822022 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 09:18:51.156141 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 09:18:51.156 UTC [12141] LOG: received smart shutdown request
2026-06-20 09:18:51.159 UTC [12141] LOG: background worker "logical replication launcher" (PID 12150) exited with exit code 1
2026-06-20 09:18:51.159 UTC [12145] LOG: shutting down
2026-06-20 09:18:51.303 UTC [12141] LOG: database system is shut down
2026-06-20T09:18:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44651 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:18:51 dist-test-slave-rq8c krb5kdc[13642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:18:51 dist-test-slave-rq8c krb5kdc[13642](info): set up 2 sockets
Jun 20 09:18:51 dist-test-slave-rq8c krb5kdc[13642](info): commencing operation
krb5kdc: starting...
W20260620 09:18:52.344385 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:18:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:18:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:18:52.461946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33217
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33217,127.0.98.61:42267,127.0.98.60:44753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:52.539758 13655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:52.539958 13655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:52.539979 13655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:52.541095 13655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:18:52.541134 13655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:52.541146 13655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:52.541158 13655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:18:52.541180 13655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:18:52.542618 13655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33217,127.0.98.61:42267,127.0.98.60:44753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:52.542840 13655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:52.543040 13655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:52.547452 13661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:52.547430 13663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:52.547434 13660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:52.547791 13655 server_base.cc:1034] running on GCE node
I20260620 09:18:52.548008 13655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:52.548261 13655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:52.549405 13655 hybrid_clock.cc:648] HybridClock initialized: now 1781947132549395 us; error 32 us; skew 500 ppm
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:52.551596 13655 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:18:52.552052 13655 webserver.cc:460] Webserver started at http://127.0.98.62:39295/ using document root <none> and password file <none>
I20260620 09:18:52.552245 13655 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:52.552290 13655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:52.552405 13655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:18:52.553382 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "5e6046efbe1342e1a5f6f930fcc79239"
format_stamp: "Formatted at 2026-06-20 09:18:52 on dist-test-slave-rq8c"
server_key: "2e2ff728207629e900e44728cc333977"
server_key_iv: "0bce4ce49c6cfac01fffc5147831ee1d"
server_key_version: "encryptionkey@0"
I20260620 09:18:52.553685 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e6046efbe1342e1a5f6f930fcc79239"
format_stamp: "Formatted at 2026-06-20 09:18:52 on dist-test-slave-rq8c"
server_key: "2e2ff728207629e900e44728cc333977"
server_key_iv: "0bce4ce49c6cfac01fffc5147831ee1d"
server_key_version: "encryptionkey@0"
I20260620 09:18:52.554802 13655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:18:52.555373 13670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:52.555517 13655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:52.555594 13655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "5e6046efbe1342e1a5f6f930fcc79239"
format_stamp: "Formatted at 2026-06-20 09:18:52 on dist-test-slave-rq8c"
server_key: "2e2ff728207629e900e44728cc333977"
server_key_iv: "0bce4ce49c6cfac01fffc5147831ee1d"
server_key_version: "encryptionkey@0"
I20260620 09:18:52.555688 13655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:52.564821 13655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:52.566560 13655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:52.566681 13655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:52.570665 13655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33217
I20260620 09:18:52.570709 13722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33217 every 8 connection(s)
I20260620 09:18:52.571018 13655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:18:52.571830 13723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:52.573369 13723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:18:52.576076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 13655
I20260620 09:18:52.576167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:18:52.576352 392 external_mini_cluster.cc:1428] Setting key 0405dd020a5c03c32ace6d02e619135d
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:52.582731 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.573991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33217 (local address 127.0.0.1:59448)
0620 09:18:52.574143 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.574188 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:52.574609 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:52.574833 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:52.574835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:52.575072 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:52.575232 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.575236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.576396 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.576397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.577071 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.577073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.577659 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.577660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.577733 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.578218 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:52.578228 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:52.579643 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:52.581417 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.581423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.581425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.581603 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.581838 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.581838 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.581839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.581909 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.582300 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:52.582303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:52.582353 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 09:18:52.582418 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 09:18:52.582539 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260620 09:18:52.582736 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.574011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.0.98.62:33217)
0620 09:18:52.574155 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.574164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.574257 (+ 93us) server_negotiation.cc:407] Connection header received
0620 09:18:52.574721 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.574723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.574738 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.574822 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.575294 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.576311 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.577151 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.577583 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.577662 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.579720 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.579728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.579729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.579736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.581335 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.581671 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.581674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.581674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.581776 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.581964 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.581965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.581965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.582239 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.582323 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.582463 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.582611 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260620 09:18:52.583647 13723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:18:52.583943 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.583854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42267 (local address 127.0.0.1:52022)
0620 09:18:52.583890 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.583927 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 09:18:52.584056 13672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42267: connect: Connection refused (error 111)
I20260620 09:18:52.584435 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.584362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42267 (local address 127.0.0.1:52034)
0620 09:18:52.584397 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.584419 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 09:18:52.584517 13723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42267: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42267: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 09:18:52.598114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42267
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33217,127.0.98.61:42267,127.0.98.60:44753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:18:52.633742 13723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } attempt: 1
I20260620 09:18:52.634294 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.634140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42267 (local address 127.0.0.1:52048)
0620 09:18:52.634202 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.634257 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 09:18:52.634672 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.634590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42267 (local address 127.0.0.1:52054)
0620 09:18:52.634628 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.634653 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 09:18:52.634765 13723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42267: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42267: connect: Connection refused (error 111)
W20260620 09:18:52.675588 13730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:52.675810 13730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:52.675832 13730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:52.676900 13730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:18:52.676932 13730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:52.676944 13730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:52.676955 13730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:18:52.676966 13730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:18:52.678376 13730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33217,127.0.98.61:42267,127.0.98.60:44753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:52.678553 13730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:52.678728 13730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:52.683580 13735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:52.683595 13736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:52.683666 13738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:52.683669 13730 server_base.cc:1034] running on GCE node
I20260620 09:18:52.684069 13730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:52.684355 13730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:52.685503 13730 hybrid_clock.cc:648] HybridClock initialized: now 1781947132685491 us; error 29 us; skew 500 ppm
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:52.691147 13730 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 09:18:52.691551 13730 webserver.cc:460] Webserver started at http://127.0.98.61:46087/ using document root <none> and password file <none>
I20260620 09:18:52.691793 13730 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:52.691838 13730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:52.691929 13730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:18:52.692978 13730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/data/instance:
uuid: "d240dbc6ba144351ad6f337970f6282e"
format_stamp: "Formatted at 2026-06-20 09:18:52 on dist-test-slave-rq8c"
server_key: "70de26f86dfee97a3651693402661a2a"
server_key_iv: "b36a975fb7cec7d114636e299b9a03fe"
server_key_version: "encryptionkey@0"
I20260620 09:18:52.693257 13730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/wal/instance:
uuid: "d240dbc6ba144351ad6f337970f6282e"
format_stamp: "Formatted at 2026-06-20 09:18:52 on dist-test-slave-rq8c"
server_key: "70de26f86dfee97a3651693402661a2a"
server_key_iv: "b36a975fb7cec7d114636e299b9a03fe"
server_key_version: "encryptionkey@0"
I20260620 09:18:52.694367 13730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:18:52.694975 13745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:52.695133 13730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:18:52.695209 13730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/wal
uuid: "d240dbc6ba144351ad6f337970f6282e"
format_stamp: "Formatted at 2026-06-20 09:18:52 on dist-test-slave-rq8c"
server_key: "70de26f86dfee97a3651693402661a2a"
server_key_iv: "b36a975fb7cec7d114636e299b9a03fe"
server_key_version: "encryptionkey@0"
I20260620 09:18:52.695263 13730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:52.702895 13723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } attempt: 2
I20260620 09:18:52.703282 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.703158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42267 (local address 127.0.0.1:52070)
0620 09:18:52.703207 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.703252 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 09:18:52.703543 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.703478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42267 (local address 127.0.0.1:52074)
0620 09:18:52.703508 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.703528 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 09:18:52.703634 13723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42267: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42267: connect: Connection refused (error 111)
I20260620 09:18:52.706372 13730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:52.708266 13730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:52.708389 13730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:52.712055 13730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42267
I20260620 09:18:52.712090 13797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42267 every 8 connection(s)
I20260620 09:18:52.712393 13730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:18:52.713207 13798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:52.714731 13798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:52.722138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 13730
I20260620 09:18:52.722236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-1/wal/instance
I20260620 09:18:52.722433 392 external_mini_cluster.cc:1428] Setting key 5af40cd247d4c3501c7b431e284c3000
I20260620 09:18:52.722785 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.715416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.0.98.62:33217)
0620 09:18:52.715474 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.715476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.715661 (+ 185us) server_negotiation.cc:407] Connection header received
0620 09:18:52.716143 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.716145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.716159 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.716210 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.716826 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.717312 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.718249 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.718457 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.718512 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.720511 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.720512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.720512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.720519 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.721572 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.721910 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.721910 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.721911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.721953 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.722167 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.722168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.722168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.722309 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.722382 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.722627 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.722705 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:18:52.722890 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.715377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33217 (local address 127.0.0.1:59462)
0620 09:18:52.715542 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.715578 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:52.716049 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:52.716279 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:52.716283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:52.716524 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:52.716742 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.716747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.717392 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.717393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.718162 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.718165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.718524 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.718525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.718644 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.719129 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:52.719137 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:52.720436 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:52.721647 (+ 1211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.721653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.721654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.721854 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.722016 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.722017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.722017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.722116 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.722374 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:52.722379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:52.722504 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 09:18:52.722604 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 09:18:52.722718 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260620 09:18:52.723376 13798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 09:18:52.729920 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.723642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42267 (local address 127.0.0.1:52088)
0620 09:18:52.723681 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.723691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:52.723725 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:52.724138 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:52.724139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:52.724259 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:52.724300 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.724301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.725085 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.725086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.725618 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.725620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.726022 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.726023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.726081 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.726438 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:52.726443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:52.727476 (+ 1033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:52.729038 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.729039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.729039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.729178 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.729343 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.729344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.729344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.729395 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.729627 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:52.729628 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:52.729663 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 09:18:52.729733 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 09:18:52.729824 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 09:18:52.730002 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.723837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52088 (local address 127.0.98.61:42267)
0620 09:18:52.723998 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.724001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.724019 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:18:52.724036 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.724037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.724047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.724117 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.724360 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.725019 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.725682 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.725965 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.726020 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.727544 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.727551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.727552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.727557 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.728963 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.729241 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.729244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.729244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.729295 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.729442 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.729443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.729443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.729573 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.729639 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.729764 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.729891 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260620 09:18:52.730508 13798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:18:52.730800 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.730690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52078)
0620 09:18:52.730731 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.730781 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 09:18:52.730921 13746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
I20260620 09:18:52.731310 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.731225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52086)
0620 09:18:52.731259 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.731288 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 09:18:52.731400 13798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44753: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 09:18:52.745687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44753
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33217,127.0.98.61:42267,127.0.98.60:44753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:18:52.780611 13798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attempt: 1
I20260620 09:18:52.781105 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.780966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52094)
0620 09:18:52.781024 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.781072 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 09:18:52.781417 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.781329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52106)
0620 09:18:52.781369 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.781396 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 09:18:52.781497 13798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44753: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
I20260620 09:18:52.794772 13723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } attempt: 3
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:18:52.801339 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.795110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42267 (local address 127.0.0.1:52096)
0620 09:18:52.795163 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.795174 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:52.795217 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:52.795442 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:52.795443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:52.795594 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:52.795703 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.795706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.796381 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.796382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.796924 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.796926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.797270 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.797271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.797340 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.797843 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:52.797848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:52.799067 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:52.800451 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.800451 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.800452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.800608 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.800769 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.800770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.800770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.800825 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.801031 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:52.801031 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:52.801079 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 09:18:52.801138 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:18:52.801237 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 09:18:52.801339 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.795298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.0.98.61:42267)
0620 09:18:52.795343 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.795345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.795351 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:18:52.795372 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.795373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.795383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.795425 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.795783 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.796299 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.796996 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.797212 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.797267 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.799148 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.799149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.799149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.799156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.800380 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.800667 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.800668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.800668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.800718 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.800872 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.800873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.800873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.800978 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.801031 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.801170 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.801240 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:18:52.801892 13723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:18:52.802191 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.802098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52120)
0620 09:18:52.802137 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.802164 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 09:18:52.802439 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.802364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52136)
0620 09:18:52.802397 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.802418 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 09:18:52.802515 13723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44753: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
W20260620 09:18:52.825071 13805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:52.825284 13805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:52.825313 13805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:52.826421 13805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:18:52.826460 13805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:52.826479 13805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:52.826495 13805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:18:52.826512 13805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:18:52.828045 13805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33217,127.0.98.61:42267,127.0.98.60:44753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:52.828280 13805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:52.828490 13805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:52.832868 13811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:52.832907 13813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:52.832890 13810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:52.832999 13805 server_base.cc:1034] running on GCE node
I20260620 09:18:52.833200 13805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:52.833446 13805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:52.833643 13723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attempt: 1
I20260620 09:18:52.833963 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.833862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52142)
0620 09:18:52.833907 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.833933 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 09:18:52.834219 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.834166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52158)
0620 09:18:52.834186 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.834206 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 09:18:52.834311 13723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44753: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
I20260620 09:18:52.834579 13805 hybrid_clock.cc:648] HybridClock initialized: now 1781947132834570 us; error 28 us; skew 500 ppm
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:52.836804 13805 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 09:18:52.837185 13805 webserver.cc:460] Webserver started at http://127.0.98.60:32879/ using document root <none> and password file <none>
I20260620 09:18:52.837378 13805 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:52.837430 13805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:52.837545 13805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:18:52.838519 13805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/data/instance:
uuid: "e4c697584a7d4c0288778b805ee868b7"
format_stamp: "Formatted at 2026-06-20 09:18:52 on dist-test-slave-rq8c"
server_key: "e71fceb1b25f40c52991d1dc7c89a952"
server_key_iv: "0277df72d3ade9711acb52cba129f551"
server_key_version: "encryptionkey@0"
I20260620 09:18:52.838835 13805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/wal/instance:
uuid: "e4c697584a7d4c0288778b805ee868b7"
format_stamp: "Formatted at 2026-06-20 09:18:52 on dist-test-slave-rq8c"
server_key: "e71fceb1b25f40c52991d1dc7c89a952"
server_key_iv: "0277df72d3ade9711acb52cba129f551"
server_key_version: "encryptionkey@0"
I20260620 09:18:52.840032 13805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:18:52.840660 13820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:52.840790 13805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:18:52.840864 13805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/wal
uuid: "e4c697584a7d4c0288778b805ee868b7"
format_stamp: "Formatted at 2026-06-20 09:18:52 on dist-test-slave-rq8c"
server_key: "e71fceb1b25f40c52991d1dc7c89a952"
server_key_iv: "0277df72d3ade9711acb52cba129f551"
server_key_version: "encryptionkey@0"
I20260620 09:18:52.840921 13805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:52.849610 13798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attempt: 2
I20260620 09:18:52.849987 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.849879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52172)
0620 09:18:52.849927 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.849959 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 09:18:52.850296 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.850210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52182)
0620 09:18:52.850253 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.850276 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 09:18:52.850381 13798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44753: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44753: connect: Connection refused (error 111)
I20260620 09:18:52.873062 13805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:52.875308 13805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:52.875420 13805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:52.879009 13805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44753
I20260620 09:18:52.879050 13872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44753 every 8 connection(s)
I20260620 09:18:52.879359 13805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:18:52.879988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 13805
I20260620 09:18:52.880076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946986834672-392-0/minicluster-data/master-2/wal/instance
I20260620 09:18:52.880245 392 external_mini_cluster.cc:1428] Setting key cd35e49b98756aef03bbfbf656a38378
I20260620 09:18:52.880368 13873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:52.882970 13873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:52.890661 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.880751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.0.98.60:44753)
0620 09:18:52.880944 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.880954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.880990 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:18:52.881491 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.881501 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.881570 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.881732 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.882311 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.883591 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.884398 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.884896 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.884977 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.887566 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.887572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.887573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.887579 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.889228 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.889535 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.889538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.889539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.889616 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.889800 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.889801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.889802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.890114 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.890251 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.890321 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.890432 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260620 09:18:52.891673 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.883689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33217 (local address 127.0.0.1:59472)
0620 09:18:52.883876 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.883910 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:52.883974 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:52.884141 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:52.884144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:52.884457 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:52.884553 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.884555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.885270 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.885272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.885986 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.885989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.886437 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.886438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.886512 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.887006 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:52.887015 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:52.888749 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:52.890361 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.890368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.890370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.890576 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.890965 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.890966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.890967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.891025 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.891321 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:52.891325 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:52.891372 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 09:18:52.891447 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:18:52.891553 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":102,"threads_started":1}
I20260620 09:18:52.891681 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.883739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.0.98.62:33217)
0620 09:18:52.883788 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.883790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.883976 (+ 186us) server_negotiation.cc:407] Connection header received
0620 09:18:52.884050 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.884051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.884063 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.884116 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.884643 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.885162 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.886076 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.886349 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.886413 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.888848 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.888849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.888850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.888857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.890289 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.890638 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.890639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.890640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.890697 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.891104 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.891105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.891105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.891256 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.891321 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.891480 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.891564 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:18:52.892490 13873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:18:52.898361 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.892701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42267 (local address 127.0.0.1:52110)
0620 09:18:52.892739 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.892749 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:52.892782 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:52.892916 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:52.892916 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:52.893045 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:52.893074 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.893076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.893647 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.893648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.894116 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.894117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.894493 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.894494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.894538 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.894856 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:52.894861 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:52.896239 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:52.897481 (+ 1242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.897481 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.897482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.897633 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.897828 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.897829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.897829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.897883 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.898096 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:52.898097 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:52.898137 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 09:18:52.898203 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:18:52.898278 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 09:18:52.898377 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.891513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.0.98.62:33217)
0620 09:18:52.891701 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.891703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.891709 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:18:52.891727 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.891728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.891738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.891802 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.892214 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.893069 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.893712 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.893920 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.893971 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.895973 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.895975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.895975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.895981 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.897375 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.897666 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.897667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.897667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.897721 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.897875 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.897875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.897876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.897995 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.898129 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.898168 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.898254 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
I20260620 09:18:52.898392 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.892772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.0.98.61:42267)
0620 09:18:52.892821 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.892823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.892831 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:18:52.892853 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.892853 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.892862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.892907 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.893136 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.893583 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.894198 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.894420 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.894483 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.896309 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.896310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.896310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.896317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.897401 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.897713 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.897714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.897715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.897769 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.897939 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.897939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.897940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.898035 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.898096 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.898237 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.898304 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:18:52.898730 13873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:18:52.904633 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.898945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52196)
0620 09:18:52.899127 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.899135 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:52.899169 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:52.899272 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:52.899273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:52.899391 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:52.899414 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.899415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.900040 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.900040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.900491 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.900492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.900805 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.900805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.900847 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.901296 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:52.901301 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:52.902381 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:52.903767 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.903768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.903768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.903914 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.904064 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.904065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.904065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.904115 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.904321 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:52.904322 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:52.904357 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 09:18:52.904416 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:18:52.904510 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260620 09:18:52.904668 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.898999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.0.98.60:44753)
0620 09:18:52.899024 (+ 25us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.899026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.899172 (+ 146us) server_negotiation.cc:407] Connection header received
0620 09:18:52.899222 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.899223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.899230 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.899272 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.899469 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.899979 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.900552 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.900752 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.900802 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.902447 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.902448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.902449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.902455 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.903707 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.903968 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.903969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.903970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.904015 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.904166 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.904167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.904167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.904272 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.904322 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.904452 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.904554 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":13}
I20260620 09:18:52.905242 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.898979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.0.98.61:42267)
0620 09:18:52.899011 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.899013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.899033 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:18:52.899050 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.899051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.899059 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.899098 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.899477 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.899965 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.900550 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.900744 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.900794 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.902994 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.902995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.902996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.903001 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.904268 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.904545 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.904546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.904547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.904592 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.904748 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.904749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.904750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.904879 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.905052 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.905089 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.905153 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:18:52.906203 13873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7: Bootstrap starting.
I20260620 09:18:52.906591 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:18:52.907189 13873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:52.907400 13873 log.cc:824] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:52.908349 13873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7: No bootstrap required, opened a new log
I20260620 09:18:52.909463 13723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attempt: 2
I20260620 09:18:52.910218 13873 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }
I20260620 09:18:52.910368 13873 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:52.910398 13873 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c697584a7d4c0288778b805ee868b7, State: Initialized, Role: FOLLOWER
I20260620 09:18:52.910512 13873 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }
I20260620 09:18:52.910796 13873 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:18:52.911116 13888 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } } }
I20260620 09:18:52.911249 13888 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:18:52.913439 13805 master_runner.cc:186] Error getting master registration for 127.0.98.62:33217: OK, instance_id { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" instance_seqno: 1781947132569617 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:18:52.916308 13805 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:18:52.916551 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.908114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.0.98.60:44753)
0620 09:18:52.908152 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.908154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.908341 (+ 187us) server_negotiation.cc:407] Connection header received
0620 09:18:52.908383 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.908384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.908393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.908441 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.909039 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.909679 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.910826 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.911407 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.911450 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.913851 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.913852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.913853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.913859 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.915319 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.915702 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.915702 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.915703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.915758 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.915956 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.915957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.915957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.916115 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.916188 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.916385 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.916466 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 09:18:52.916821 13899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:18:52.916903 13899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:18:52.917898 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.909676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52210)
0620 09:18:52.909767 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.909776 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:52.909809 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:52.910147 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:52.910148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:52.910301 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:52.910335 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.910337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.911576 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.911577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.912330 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.912332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.912787 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.912788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.912843 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.913339 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:52.913344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:52.914763 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:52.916881 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.916882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.916883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.917095 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.917305 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.917306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.917306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.917358 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.917620 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:52.917621 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:52.917655 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:18:52.917708 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 09:18:52.917808 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 09:18:52.918062 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.907921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.0.98.62:33217)
0620 09:18:52.907964 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.907966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.908067 (+ 101us) server_negotiation.cc:407] Connection header received
0620 09:18:52.908158 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.908159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.908171 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.908201 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.908842 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.909569 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.910438 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.910717 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.910758 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.912243 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.912244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.912245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.912252 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.916971 (+ 4719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.917261 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.917262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.917263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.917317 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.917499 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.917499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.917500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.917612 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.917705 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.917915 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.917985 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:18:52.918700 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.909868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.0.98.60:44753)
0620 09:18:52.910019 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.910021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.910028 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:18:52.910046 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.910046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.910055 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.910527 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.910652 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.911484 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.912415 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.912689 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.912947 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.914881 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.914883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.914883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.914890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.916796 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.917177 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.917178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.917178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.917247 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.917423 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.917424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.917424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.917545 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.917598 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.918569 (+ 971us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.918633 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260620 09:18:52.919963 13723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239: Bootstrap starting.
I20260620 09:18:52.920662 13723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:52.920884 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.912141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.0.98.61:42267)
0620 09:18:52.912177 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.912179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.912186 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:18:52.912281 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.912283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.912293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.912355 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.913344 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.914304 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.915957 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.916233 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.916270 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.918118 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.918119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.918120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.918127 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.919703 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.920045 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.920046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.920046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.920105 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.920290 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.920291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.920291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.920417 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.920518 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.920626 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.920703 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:18:52.920926 13723 log.cc:824] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:52.921742 13723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239: No bootstrap required, opened a new log
I20260620 09:18:52.923079 13723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }
I20260620 09:18:52.923188 13723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:52.923228 13723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6046efbe1342e1a5f6f930fcc79239, State: Initialized, Role: FOLLOWER
I20260620 09:18:52.923344 13723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }
I20260620 09:18:52.923684 13723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:18:52.923610 13902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } } }
I20260620 09:18:52.923738 13902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 09:18:52.925565 13913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:18:52.925621 13913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:18:52.926277 13655 master_runner.cc:186] Error getting master registration for 127.0.98.61:42267: OK, instance_id { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" instance_seqno: 1781947132710964 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:18:52.928539 13655 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:18:52.941537 13798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } attempt: 3
Jun 20 09:18:52 dist-test-slave-rq8c krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:18:52.949630 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.941812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44753 (local address 127.0.0.1:52214)
0620 09:18:52.941863 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:52.941873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:52.941907 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:52.942054 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:52.942055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:52.942247 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:52.942300 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.942302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.943258 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.943259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.944135 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:52.944138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.944684 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.944685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:52.944750 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.945256 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:52.945261 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:52.946737 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:52.948575 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.948576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.948576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.948784 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.948975 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:52.948976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:52.948976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:52.949035 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:52.949307 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:52.949308 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:52.949355 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 09:18:52.949423 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 09:18:52.949522 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 09:18:52.949630 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:52.941882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.0.98.60:44753)
0620 09:18:52.941942 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:52.941944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:52.941953 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:18:52.941975 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:52.941975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:52.941985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:52.942045 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:52.942371 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.943179 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.944218 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:52.944598 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:52.944662 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:52.946818 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:52.946820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:52.946820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:52.946828 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:52.948495 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.948855 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.948856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.948856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.948915 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:52.949093 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:52.949094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:52.949095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:52.949246 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:52.949312 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:52.949453 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 09:18:52.949536 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:18:52.951542 13798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e: Bootstrap starting.
I20260620 09:18:52.952309 13798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:52.952603 13798 log.cc:824] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:52.953450 13798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e: No bootstrap required, opened a new log
I20260620 09:18:52.955399 13798 raft_consensus.cc:348] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }
I20260620 09:18:52.955534 13798 raft_consensus.cc:374] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:52.955565 13798 raft_consensus.cc:729] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d240dbc6ba144351ad6f337970f6282e, State: Initialized, Role: FOLLOWER
I20260620 09:18:52.955719 13798 consensus_queue.cc:260] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }
I20260620 09:18:52.956029 13798 sys_catalog.cc:564] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:18:52.955987 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } } }
I20260620 09:18:52.956169 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [sys.catalog]: This master's current role is: FOLLOWER
W20260620 09:18:52.957800 13927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:18:52.957896 13927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 09:18:52.958928 13751 tablet.cc:2369] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:18:52.959789 13730 master_runner.cc:263] No leader master found from master d240dbc6ba144351ad6f337970f6282e
I20260620 09:18:52.959836 13730 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:18:53.006587 13888 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:18:53.006731 13888 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }
I20260620 09:18:53.006999 13888 leader_election.cc:288] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e6046efbe1342e1a5f6f930fcc79239 (127.0.98.62:33217), d240dbc6ba144351ad6f337970f6282e (127.0.98.61:42267)
I20260620 09:18:53.007510 13772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4c697584a7d4c0288778b805ee868b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d240dbc6ba144351ad6f337970f6282e" is_pre_election: true
I20260620 09:18:53.007717 13772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c697584a7d4c0288778b805ee868b7 in term 0.
I20260620 09:18:53.007920 13822 leader_election.cc:302] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d240dbc6ba144351ad6f337970f6282e, e4c697584a7d4c0288778b805ee868b7; no voters:
I20260620 09:18:53.008147 13697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4c697584a7d4c0288778b805ee868b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6046efbe1342e1a5f6f930fcc79239" is_pre_election: true
I20260620 09:18:53.008318 13697 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c697584a7d4c0288778b805ee868b7 in term 0.
I20260620 09:18:53.008567 13888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:18:53.008631 13888 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:18:53.008658 13888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:53.009559 13888 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }
I20260620 09:18:53.009686 13888 leader_election.cc:288] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [CANDIDATE]: Term 1 election: Requested vote from peers 5e6046efbe1342e1a5f6f930fcc79239 (127.0.98.62:33217), d240dbc6ba144351ad6f337970f6282e (127.0.98.61:42267)
I20260620 09:18:53.009943 13772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4c697584a7d4c0288778b805ee868b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d240dbc6ba144351ad6f337970f6282e"
I20260620 09:18:53.010023 13772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:53.010886 13772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c697584a7d4c0288778b805ee868b7 in term 1.
I20260620 09:18:53.011045 13822 leader_election.cc:302] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d240dbc6ba144351ad6f337970f6282e, e4c697584a7d4c0288778b805ee868b7; no voters:
I20260620 09:18:53.011142 13697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4c697584a7d4c0288778b805ee868b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6046efbe1342e1a5f6f930fcc79239"
I20260620 09:18:53.011229 13697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:53.012025 13697 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c697584a7d4c0288778b805ee868b7 in term 1.
I20260620 09:18:53.013044 13888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:18:53.013212 13888 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [term 1 LEADER]: Becoming Leader. State: Replica: e4c697584a7d4c0288778b805ee868b7, State: Running, Role: LEADER
I20260620 09:18:53.013300 13888 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } }
I20260620 09:18:53.013654 13888 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4c697584a7d4c0288778b805ee868b7. Latest consensus state: current_term: 1 leader_uuid: "e4c697584a7d4c0288778b805ee868b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } } }
I20260620 09:18:53.013725 13888 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:53.013911 13931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:18:53.014091 13931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:18:53.015939 13772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [term 1 FOLLOWER]: Refusing update from remote peer e4c697584a7d4c0288778b805ee868b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:18:53.016223 13888 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:18:53.017412 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4c697584a7d4c0288778b805ee868b7. Latest consensus state: current_term: 1 leader_uuid: "e4c697584a7d4c0288778b805ee868b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } } }
I20260620 09:18:53.017513 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:18:53.017843 13697 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [term 1 FOLLOWER]: Refusing update from remote peer e4c697584a7d4c0288778b805ee868b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:18:53.018082 13888 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:18:53.019222 13902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4c697584a7d4c0288778b805ee868b7. Latest consensus state: current_term: 1 leader_uuid: "e4c697584a7d4c0288778b805ee868b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } } }
I20260620 09:18:53.019330 13902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:18:53.020601 13932 mvcc.cc:205] Tried to move back new op lower bound from 7298855456831225856 to 7298855456823091200. Current Snapshot: MvccSnapshot[applied={T|T < 7298855456831225856}]
I20260620 09:18:53.020843 13888 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4c697584a7d4c0288778b805ee868b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } } }
I20260620 09:18:53.020917 13888 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:53.021340 13902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4c697584a7d4c0288778b805ee868b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } } }
I20260620 09:18:53.021425 13902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:18:53.021667 13935 mvcc.cc:205] Tried to move back new op lower bound from 7298855456831225856 to 7298855456823091200. Current Snapshot: MvccSnapshot[applied={T|T < 7298855456831225856}]
I20260620 09:18:53.022425 13888 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4c697584a7d4c0288778b805ee868b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } } }
I20260620 09:18:53.022502 13888 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:53.022828 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4c697584a7d4c0288778b805ee868b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6046efbe1342e1a5f6f930fcc79239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } peers { permanent_uuid: "d240dbc6ba144351ad6f337970f6282e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42267 } } peers { permanent_uuid: "e4c697584a7d4c0288778b805ee868b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44753 } } }
I20260620 09:18:53.022923 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:18:53.023662 13933 mvcc.cc:205] Tried to move back new op lower bound from 7298855456831225856 to 7298855456823091200. Current Snapshot: MvccSnapshot[applied={T|T < 7298855456831225856}]
I20260620 09:18:53.025533 13931 catalog_manager.cc:1347] Generated new cluster ID: 16cf199313ae4dc59c47030b3eee6181
I20260620 09:18:53.025584 13931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:18:53.051740 13931 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:18:53.052486 13931 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:18:53.058290 13931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4c697584a7d4c0288778b805ee868b7: Generated new TSK 0
I20260620 09:18:53.058542 13931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:18:53.926071 13913 catalog_manager.cc:1259] Loaded cluster ID: 16cf199313ae4dc59c47030b3eee6181
I20260620 09:18:53.926154 13913 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239: loading cluster ID for follower catalog manager: success
I20260620 09:18:53.927008 13913 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239: acquiring CA information for follower catalog manager: success
I20260620 09:18:53.927304 13913 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e6046efbe1342e1a5f6f930fcc79239: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:18:53.958319 13927 catalog_manager.cc:1259] Loaded cluster ID: 16cf199313ae4dc59c47030b3eee6181
I20260620 09:18:53.958401 13927 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e: loading cluster ID for follower catalog manager: success
I20260620 09:18:53.959306 13927 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e: acquiring CA information for follower catalog manager: success
I20260620 09:18:53.959645 13927 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d240dbc6ba144351ad6f337970f6282e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:18:54.105348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 13655
I20260620 09:18:54.110117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 13730
I20260620 09:18:54.115767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 13805
2026-06-20T09:18:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2729 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:18:54 dist-test-slave-rq8c krb5kdc[13946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:18:54 dist-test-slave-rq8c krb5kdc[13946](info): set up 2 sockets
Jun 20 09:18:54 dist-test-slave-rq8c krb5kdc[13946](info): commencing operation
krb5kdc: starting...
W20260620 09:18:55.063283 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:18:55 dist-test-slave-rq8c krb5kdc[13946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:18:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:18:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:18:55.170622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45489
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:55.248126 13959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:55.248325 13959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:55.248354 13959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:55.249480 13959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:18:55.249521 13959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:55.249538 13959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:55.249555 13959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:18:55.249570 13959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:18:55.251001 13959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:55.251228 13959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:55.251434 13959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:55.256070 13965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:55.256114 13964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:55.256148 13959 server_base.cc:1034] running on GCE node
W20260620 09:18:55.256116 13967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:55.256536 13959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:55.256801 13959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:55.257966 13959 hybrid_clock.cc:648] HybridClock initialized: now 1781947135257952 us; error 30 us; skew 500 ppm
Jun 20 09:18:55 dist-test-slave-rq8c krb5kdc[13946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:55.260362 13959 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:18:55.260767 13959 webserver.cc:460] Webserver started at http://127.0.98.62:39615/ using document root <none> and password file <none>
I20260620 09:18:55.260978 13959 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:55.261026 13959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:55.261147 13959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:18:55.262142 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "7f45f70768824254a4be8373e33d3e30"
format_stamp: "Formatted at 2026-06-20 09:18:55 on dist-test-slave-rq8c"
server_key: "1fb95bf607ce081491fe696f24718ae6"
server_key_iv: "65785e25f4040446330d001c45f56139"
server_key_version: "encryptionkey@0"
I20260620 09:18:55.262450 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "7f45f70768824254a4be8373e33d3e30"
format_stamp: "Formatted at 2026-06-20 09:18:55 on dist-test-slave-rq8c"
server_key: "1fb95bf607ce081491fe696f24718ae6"
server_key_iv: "65785e25f4040446330d001c45f56139"
server_key_version: "encryptionkey@0"
I20260620 09:18:55.263644 13959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:18:55.264310 13974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:55.264457 13959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:18:55.264544 13959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "7f45f70768824254a4be8373e33d3e30"
format_stamp: "Formatted at 2026-06-20 09:18:55 on dist-test-slave-rq8c"
server_key: "1fb95bf607ce081491fe696f24718ae6"
server_key_iv: "65785e25f4040446330d001c45f56139"
server_key_version: "encryptionkey@0"
I20260620 09:18:55.264608 13959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:55.276602 13959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:55.278646 13959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:55.278771 13959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:55.282302 13959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45489
I20260620 09:18:55.282337 14026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45489 every 8 connection(s)
I20260620 09:18:55.282634 13959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:18:55.283373 14027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:55.284623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 13959
I20260620 09:18:55.284710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:18:55.284869 392 external_mini_cluster.cc:1428] Setting key 359371dc2de4223ebbd443450e5ba0cc
I20260620 09:18:55.285766 14027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30: Bootstrap starting.
I20260620 09:18:55.286386 14027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:55.286702 14027 log.cc:824] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:55.287340 14027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30: No bootstrap required, opened a new log
I20260620 09:18:55.289552 14027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f45f70768824254a4be8373e33d3e30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45489 } }
I20260620 09:18:55.289716 14027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:55.289760 14027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f45f70768824254a4be8373e33d3e30, State: Initialized, Role: FOLLOWER
I20260620 09:18:55.289891 14027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f45f70768824254a4be8373e33d3e30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45489 } }
I20260620 09:18:55.289971 14027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:18:55.290009 14027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:18:55.290057 14027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:55.290748 14027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f45f70768824254a4be8373e33d3e30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45489 } }
I20260620 09:18:55.290836 14027 leader_election.cc:302] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f45f70768824254a4be8373e33d3e30; no voters:
I20260620 09:18:55.290980 14027 leader_election.cc:288] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:18:55.291013 14032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:18:55.291165 14032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [term 1 LEADER]: Becoming Leader. State: Replica: 7f45f70768824254a4be8373e33d3e30, State: Running, Role: LEADER
I20260620 09:18:55.291234 14027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 09:18:55 dist-test-slave-rq8c krb5kdc[13946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:55.291297 14032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f45f70768824254a4be8373e33d3e30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45489 } }
I20260620 09:18:55.291651 14032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f45f70768824254a4be8373e33d3e30. Latest consensus state: current_term: 1 leader_uuid: "7f45f70768824254a4be8373e33d3e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f45f70768824254a4be8373e33d3e30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45489 } } }
I20260620 09:18:55.291720 14032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:55.291710 14034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f45f70768824254a4be8373e33d3e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f45f70768824254a4be8373e33d3e30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45489 } } }
I20260620 09:18:55.291800 14034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:55.291962 14039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:18:55.292753 14039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:18:55.294381 14039 catalog_manager.cc:1347] Generated new cluster ID: 89fb0e6285de483aba88c0bc12c952a3
I20260620 09:18:55.294430 14039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:18:55.295444 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:55.285314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.0.98.62:45489)
0620 09:18:55.285466 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:55.285474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:55.285498 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:18:55.285897 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:55.285903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:55.285950 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:55.286062 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:55.286769 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:55.287719 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:55.288746 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:55.289129 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:55.289196 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:55.291815 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:55.291819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:55.291820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:55.291831 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:55.293887 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:55.294232 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:55.294235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:55.294235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:55.294339 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:55.294551 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:55.294552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:55.294552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:55.294855 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:55.295014 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:55.295115 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 09:18:55.295248 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260620 09:18:55.296305 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:18:55.302332 14039 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:18:55.302810 14039 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:18:55.305476 14039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f45f70768824254a4be8373e33d3e30: Generated new TSK 0
I20260620 09:18:55.305619 14039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:18:55.350189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 13959
2026-06-20T09:18:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1230 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:18:55 dist-test-slave-rq8c krb5kdc[14054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:18:55 dist-test-slave-rq8c krb5kdc[14054](info): set up 2 sockets
Jun 20 09:18:55 dist-test-slave-rq8c krb5kdc[14054](info): commencing operation
krb5kdc: starting...
W20260620 09:18:56.287384 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:18:56 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:18:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:18:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:18:56.395989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35695
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:56.474323 14067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:56.474515 14067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:56.474535 14067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:56.475669 14067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:18:56.475705 14067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:56.475718 14067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:56.475729 14067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:18:56.475739 14067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:18:56.477149 14067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:56.477346 14067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:56.477519 14067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:18:56.482065 14067 server_base.cc:1034] running on GCE node
W20260620 09:18:56.481997 14072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:56.481985 14073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:56.481986 14075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:56.482831 14067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:56.483086 14067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:56.484220 14067 hybrid_clock.cc:648] HybridClock initialized: now 1781947136484209 us; error 28 us; skew 500 ppm
Jun 20 09:18:56 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:56.486443 14067 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:18:56.486855 14067 webserver.cc:460] Webserver started at http://127.0.98.62:36573/ using document root <none> and password file <none>
I20260620 09:18:56.487053 14067 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:56.487094 14067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:56.487216 14067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:18:56.488289 14067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "a8c65c5d094b4ae19c76864cbbac8ea9"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "3d29bbee7f8c3c8cdb81407b396f3cbf"
server_key_iv: "5f72aaf9d98e5a2d9dced94635212d71"
server_key_version: "encryptionkey@0"
I20260620 09:18:56.488615 14067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "a8c65c5d094b4ae19c76864cbbac8ea9"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "3d29bbee7f8c3c8cdb81407b396f3cbf"
server_key_iv: "5f72aaf9d98e5a2d9dced94635212d71"
server_key_version: "encryptionkey@0"
I20260620 09:18:56.489771 14067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:18:56.490409 14082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:56.490553 14067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:18:56.490630 14067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "a8c65c5d094b4ae19c76864cbbac8ea9"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "3d29bbee7f8c3c8cdb81407b396f3cbf"
server_key_iv: "5f72aaf9d98e5a2d9dced94635212d71"
server_key_version: "encryptionkey@0"
I20260620 09:18:56.490692 14067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:56.518383 14067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:56.520262 14067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:56.520390 14067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:56.524089 14067 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35695
I20260620 09:18:56.524132 14134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35695 every 8 connection(s)
I20260620 09:18:56.524453 14067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:18:56.525357 14135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:56.527562 14135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Bootstrap starting.
I20260620 09:18:56.528126 14135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:56.528376 14135 log.cc:824] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:56.529004 14135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: No bootstrap required, opened a new log
I20260620 09:18:56.530367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 14067
I20260620 09:18:56.530452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:18:56.530627 392 external_mini_cluster.cc:1428] Setting key 170391c455a616a6f1ab6a5113451695
I20260620 09:18:56.530579 14135 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260620 09:18:56.530725 14135 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:56.530766 14135 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c65c5d094b4ae19c76864cbbac8ea9, State: Initialized, Role: FOLLOWER
I20260620 09:18:56.530900 14135 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260620 09:18:56.530977 14135 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:18:56.531026 14135 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:18:56.531077 14135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:56.532099 14135 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260620 09:18:56.532208 14135 leader_election.cc:302] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8c65c5d094b4ae19c76864cbbac8ea9; no voters:
I20260620 09:18:56.532356 14135 leader_election.cc:288] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:18:56.532377 14140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:18:56.532475 14140 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 1 LEADER]: Becoming Leader. State: Replica: a8c65c5d094b4ae19c76864cbbac8ea9, State: Running, Role: LEADER
I20260620 09:18:56.532608 14140 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260620 09:18:56.532604 14135 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:18:56.532960 14142 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8c65c5d094b4ae19c76864cbbac8ea9. Latest consensus state: current_term: 1 leader_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } } }
I20260620 09:18:56.533171 14142 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:56.533114 14141 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } } }
I20260620 09:18:56.533473 14141 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:56.533524 14144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:18:56.534081 14144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:18:56.535548 14144 catalog_manager.cc:1347] Generated new cluster ID: 54d10f55162f4fc196d0ef50037f3c15
I20260620 09:18:56.535663 14144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:18:56 dist-test-slave-rq8c krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:56.540621 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:56.531040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60586 (local address 127.0.98.62:35695)
0620 09:18:56.531221 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:56.531229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:56.531258 (+ 29us) server_negotiation.cc:407] Connection header received
0620 09:18:56.531358 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:56.531363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:56.531412 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:56.531518 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:56.532338 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.533650 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.534680 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.535245 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.535359 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:56.537523 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:56.537527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:56.537528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:56.537539 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:56.539047 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:56.539362 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:56.539366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:56.539366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:56.539475 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:56.539670 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:56.539671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:56.539671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:56.539920 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:56.540088 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:56.540170 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 09:18:56.540315 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260620 09:18:56.548574 14144 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:18:56.549170 14144 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:18:56.557847 14144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Generated new TSK 0
I20260620 09:18:56.558032 14144 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:18:56.584682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35695
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:56.666002 14161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:56.666249 14161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:56.666283 14161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:56.667490 14161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:56.667531 14161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:56.667557 14161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:18:56.669050 14161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:56.669265 14161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:56.669538 14161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:56.674517 14169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:56.674526 14166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:56.674532 14167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:56.674695 14161 server_base.cc:1034] running on GCE node
I20260620 09:18:56.674860 14161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:56.675086 14161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:56.676218 14161 hybrid_clock.cc:648] HybridClock initialized: now 1781947136676205 us; error 31 us; skew 500 ppm
Jun 20 09:18:56 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:56.678517 14161 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:18:56.678939 14161 webserver.cc:460] Webserver started at http://127.0.98.1:45741/ using document root <none> and password file <none>
I20260620 09:18:56.679141 14161 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:56.679184 14161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:56.679306 14161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:18:56.680361 14161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "c61fc3e9229f41b0b30a1e2e5adebafe"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "bc522f3779c136b5fc94421ac0d6c3d1"
server_key_iv: "4ef173d5c55294b062c64ea1a817c78b"
server_key_version: "encryptionkey@0"
I20260620 09:18:56.680670 14161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c61fc3e9229f41b0b30a1e2e5adebafe"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "bc522f3779c136b5fc94421ac0d6c3d1"
server_key_iv: "4ef173d5c55294b062c64ea1a817c78b"
server_key_version: "encryptionkey@0"
I20260620 09:18:56.681921 14161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:18:56.682636 14176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:56.682818 14161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:18:56.682906 14161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "c61fc3e9229f41b0b30a1e2e5adebafe"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "bc522f3779c136b5fc94421ac0d6c3d1"
server_key_iv: "4ef173d5c55294b062c64ea1a817c78b"
server_key_version: "encryptionkey@0"
I20260620 09:18:56.682968 14161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:56.692353 14161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:56.694330 14161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:56.694454 14161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:56.694655 14161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:56.694913 14161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:56.694942 14161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:56.694972 14161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:56.694988 14161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:56.700258 14161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35131
I20260620 09:18:56.700341 14289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35131 every 8 connection(s)
I20260620 09:18:56.700628 14161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:18:56 dist-test-slave-rq8c krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:56.709393 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:56.701370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41195 (local address 127.0.98.62:35695)
0620 09:18:56.701445 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:56.701447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:56.701638 (+ 191us) server_negotiation.cc:407] Connection header received
0620 09:18:56.702186 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:56.702187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:56.702202 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:56.702270 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:56.703202 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.703739 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.704745 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.704947 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.705017 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:56.707160 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:56.707161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:56.707162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:56.707168 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:56.708356 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:56.708696 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:56.708697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:56.708697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:56.708742 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:56.708917 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:56.708918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:56.708918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:56.709009 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:56.709053 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:56.709250 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 09:18:56.709307 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:18:56.709523 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:56.701334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:41195)
0620 09:18:56.701512 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:56.701555 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:56.702071 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:56.702334 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:56.702337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:56.702903 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:56.703099 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:56.703104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.703828 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.703829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:56.704652 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:56.704655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.705051 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.705053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:56.705180 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:56.705723 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:56.705732 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:56.707069 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:56.708430 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:56.708435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:56.708437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:56.708642 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:56.708785 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:56.708785 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:56.708785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:56.708875 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:56.709049 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:56.709052 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:56.709160 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 09:18:56.709233 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:18:56.709347 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260620 09:18:56.709834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 14161
I20260620 09:18:56.709941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:18:56.710008 14290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35695
I20260620 09:18:56.710106 14290 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:56.710148 392 external_mini_cluster.cc:1428] Setting key 9678051d53eb1c9fd6be6830eafce9fb
I20260620 09:18:56.710302 14290 heartbeater.cc:511] Master 127.0.98.62:35695 requested a full tablet report, sending...
I20260620 09:18:56.710846 14098 ts_manager.cc:194] Registered new tserver with Master: c61fc3e9229f41b0b30a1e2e5adebafe (127.0.98.1:35131)
I20260620 09:18:56.711439 14098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41195
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:18:56.732544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35695
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:56.812083 14295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:56.812270 14295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:56.812292 14295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:56.813381 14295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:56.813413 14295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:56.813437 14295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:18:56.814814 14295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:56.814985 14295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:56.815152 14295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:56.819885 14303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:56.819994 14295 server_base.cc:1034] running on GCE node
W20260620 09:18:56.819976 14300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:56.819885 14301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:56.820340 14295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:56.820570 14295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:56.821749 14295 hybrid_clock.cc:648] HybridClock initialized: now 1781947136821723 us; error 28 us; skew 500 ppm
Jun 20 09:18:56 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:56.824383 14295 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:18:56.824841 14295 webserver.cc:460] Webserver started at http://127.0.98.2:36293/ using document root <none> and password file <none>
I20260620 09:18:56.825042 14295 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:56.825091 14295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:56.825215 14295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:18:56.826258 14295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "770d42b044b94d56a1ba645952e4496a"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "4168d476e84ac55beaf190c63a151ed5"
server_key_iv: "1302614463de437f55a7af17f9cc9cd5"
server_key_version: "encryptionkey@0"
I20260620 09:18:56.826598 14295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "770d42b044b94d56a1ba645952e4496a"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "4168d476e84ac55beaf190c63a151ed5"
server_key_iv: "1302614463de437f55a7af17f9cc9cd5"
server_key_version: "encryptionkey@0"
I20260620 09:18:56.827915 14295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:18:56.828567 14310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:56.828707 14295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:18:56.828785 14295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "770d42b044b94d56a1ba645952e4496a"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "4168d476e84ac55beaf190c63a151ed5"
server_key_iv: "1302614463de437f55a7af17f9cc9cd5"
server_key_version: "encryptionkey@0"
I20260620 09:18:56.828841 14295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:56.839890 14295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:56.841538 14295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:56.841666 14295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:56.841867 14295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:56.842149 14295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:56.842181 14295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:56.842211 14295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:56.842227 14295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:56.847600 14295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38951
I20260620 09:18:56.847668 14423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38951 every 8 connection(s)
I20260620 09:18:56.848014 14295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:18:56 dist-test-slave-rq8c krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:56.856138 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:56.848671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43951 (local address 127.0.98.62:35695)
0620 09:18:56.848736 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:56.848738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:56.848911 (+ 173us) server_negotiation.cc:407] Connection header received
0620 09:18:56.849373 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:56.849374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:56.849383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:56.849425 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:56.850288 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.850765 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.851711 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.851907 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.851953 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:56.853882 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:56.853884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:56.853884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:56.853891 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:56.854982 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:56.855308 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:56.855309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:56.855309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:56.855365 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:56.855566 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:56.855567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:56.855567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:56.855724 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:56.855778 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:56.856004 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 09:18:56.856068 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:18:56.856295 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:56.848643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:43951)
0620 09:18:56.848797 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:56.848832 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:56.849288 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:56.849480 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:56.849484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:56.850033 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:56.850225 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:56.850230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.850824 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.850824 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:56.851612 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:56.851635 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.851978 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.851979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:56.852082 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:56.852514 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:56.852522 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:56.853798 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:56.855047 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:56.855053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:56.855055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:56.855254 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:56.855408 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:56.855408 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:56.855408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:56.855493 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:56.855782 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:56.855785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:56.855907 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 09:18:56.855985 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:18:56.856099 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260620 09:18:56.856730 14424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35695
I20260620 09:18:56.856817 14424 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:56.856978 14424 heartbeater.cc:511] Master 127.0.98.62:35695 requested a full tablet report, sending...
I20260620 09:18:56.857029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 14295
I20260620 09:18:56.857123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:18:56.857304 392 external_mini_cluster.cc:1428] Setting key 6b42fe5cc260ef71c0dbbaec103f34ff
I20260620 09:18:56.857322 14098 ts_manager.cc:194] Registered new tserver with Master: 770d42b044b94d56a1ba645952e4496a (127.0.98.2:38951)
I20260620 09:18:56.857683 14098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43951
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:18:56.878695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35695
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:56.957693 14429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:56.957893 14429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:56.957917 14429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:56.958994 14429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:56.959028 14429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:56.959053 14429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:18:56.960482 14429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:56.960662 14429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:56.960839 14429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:56.965456 14434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:56.965476 14435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:56.965498 14437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:56.965694 14429 server_base.cc:1034] running on GCE node
I20260620 09:18:56.965958 14429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:56.966176 14429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:56.967309 14429 hybrid_clock.cc:648] HybridClock initialized: now 1781947136967292 us; error 34 us; skew 500 ppm
Jun 20 09:18:56 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:56.969599 14429 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:18:56.970012 14429 webserver.cc:460] Webserver started at http://127.0.98.3:35959/ using document root <none> and password file <none>
I20260620 09:18:56.970204 14429 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:56.970250 14429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:56.970368 14429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:18:56.971385 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "6fa09c96387c49359a077279b4a33a5c"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "a1e329a15927fef8080c816bb6d6e812"
server_key_iv: "43f58a2fc1c7030e95866f7a96767f30"
server_key_version: "encryptionkey@0"
I20260620 09:18:56.971735 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6fa09c96387c49359a077279b4a33a5c"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "a1e329a15927fef8080c816bb6d6e812"
server_key_iv: "43f58a2fc1c7030e95866f7a96767f30"
server_key_version: "encryptionkey@0"
I20260620 09:18:56.972965 14429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:18:56.973588 14444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:56.973719 14429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:18:56.973798 14429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "6fa09c96387c49359a077279b4a33a5c"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "a1e329a15927fef8080c816bb6d6e812"
server_key_iv: "43f58a2fc1c7030e95866f7a96767f30"
server_key_version: "encryptionkey@0"
I20260620 09:18:56.973855 14429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:56.982350 14429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:56.984066 14429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:56.984200 14429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:56.984385 14429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:56.984654 14429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:56.984685 14429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:56.984712 14429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:56.984736 14429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:56.990106 14429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41293
I20260620 09:18:56.990151 14557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41293 every 8 connection(s)
I20260620 09:18:56.990437 14429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:18:56.993059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 14429
I20260620 09:18:56.993139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:18:56.993314 392 external_mini_cluster.cc:1428] Setting key 8bc9038b730dd4d22226ab419cfcc238
Jun 20 09:18:56 dist-test-slave-rq8c krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:56.998981 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:56.991090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32793 (local address 127.0.98.62:35695)
0620 09:18:56.991142 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:56.991144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:56.991319 (+ 175us) server_negotiation.cc:407] Connection header received
0620 09:18:56.991877 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:56.991878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:56.991888 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:56.991943 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:56.992843 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.993417 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.994617 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.994805 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.994857 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:56.996765 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:56.996766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:56.996767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:56.996773 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:56.997991 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:56.998311 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:56.998312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:56.998312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:56.998357 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:56.998536 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:56.998536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:56.998537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:56.998626 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:56.998669 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:56.998861 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 09:18:56.998918 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:18:56.999141 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:56.991072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:32793)
0620 09:18:56.991212 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:56.991245 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:56.991790 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:56.992015 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:56.992019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:56.992569 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:56.992752 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:56.992758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.993491 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.993491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:56.994553 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:56.994556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:56.994861 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:56.994862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:56.994954 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:56.995351 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:56.995358 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:56.996681 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:56.998058 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:56.998063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:56.998065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:56.998261 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:56.998408 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:56.998409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:56.998409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:56.998497 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:56.998665 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:56.998668 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:56.998772 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 09:18:56.998843 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 09:18:56.998954 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260620 09:18:56.999593 14558 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35695
I20260620 09:18:56.999699 14558 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:56.999861 14558 heartbeater.cc:511] Master 127.0.98.62:35695 requested a full tablet report, sending...
I20260620 09:18:57.000190 14098 ts_manager.cc:194] Registered new tserver with Master: 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293)
I20260620 09:18:57.000545 14098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32793
I20260620 09:18:57.004900 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:18:57.011545 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.006197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.0.98.62:35695)
0620 09:18:57.006249 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.006251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.006391 (+ 140us) server_negotiation.cc:407] Connection header received
0620 09:18:57.006431 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.006432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.006442 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.006493 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:57.007004 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.007471 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.008237 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.008435 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.008486 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.009493 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:57.009494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:57.009495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:57.009501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:57.010571 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:57.010848 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:57.010849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:57.010849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:57.010899 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:57.011052 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:57.011053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:57.011053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:57.011190 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:57.011272 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.011390 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.011458 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 09:18:57.012920 14098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:18:57.014381 14098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:18:57.020393 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.016541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35131 (local address 127.0.0.1:59816)
0620 09:18:57.016625 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.016633 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.016671 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.017249 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.017250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.017252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:57.017296 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.017298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.018429 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.018429 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.019441 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.019443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.020101 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.020102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.020185 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.020199 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.020263 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.020315 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1,"thread_start_us":54,"threads_started":1}
I20260620 09:18:57.020406 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.016881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.0.98.1:35131)
0620 09:18:57.017041 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.017046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.017055 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:18:57.017092 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.017094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.017108 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.017174 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:57.017363 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.018350 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.019507 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.020032 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.020094 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.020159 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.020281 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.020318 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260620 09:18:57.020474 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.016426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.0.98.3:41293)
0620 09:18:57.016620 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.016625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.016635 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:18:57.016670 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.016673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.016688 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.016769 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:57.016909 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.017914 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.019322 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.020065 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.020134 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.020182 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.020359 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.020395 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":96,"threads_started":1}
I20260620 09:18:57.020812 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.016335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41293 (local address 127.0.0.1:35786)
0620 09:18:57.016446 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.016489 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.016555 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.016771 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.016774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.016776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:57.016842 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.016843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.018017 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.018018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.019246 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.019249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.020213 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.020214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.020275 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.020278 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.020699 (+ 421us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.020743 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":63,"threads_started":1}
I20260620 09:18:57.021113 14224 tablet_service.cc:1467] Processing CreateTablet for tablet 9d77175947e14a7b9aa0944520f30ebd (DEFAULT_TABLE table=test-table [id=41965de5be844a49a46388e3b929dec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:18:57.021461 14224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d77175947e14a7b9aa0944520f30ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:57.021569 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.016673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38951 (local address 127.0.0.1:40890)
0620 09:18:57.017052 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.017062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.017108 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.018233 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.018234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.018235 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:57.018265 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.018266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.019210 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.019211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.020306 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.020308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.021397 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.021398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.021462 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.021465 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.021496 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.021525 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":69,"threads_started":1}
I20260620 09:18:57.021472 14492 tablet_service.cc:1467] Processing CreateTablet for tablet 9d77175947e14a7b9aa0944520f30ebd (DEFAULT_TABLE table=test-table [id=41965de5be844a49a46388e3b929dec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:18:57.021742 14492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d77175947e14a7b9aa0944520f30ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:57.021770 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.016783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.0.98.2:38951)
0620 09:18:57.018090 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.018094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.018102 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:18:57.018137 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.018139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.018153 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.018219 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:57.018320 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.019130 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.020383 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.021075 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.021302 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.021364 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.021654 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.021698 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1269,"thread_start_us":97,"threads_started":1}
I20260620 09:18:57.022262 14358 tablet_service.cc:1467] Processing CreateTablet for tablet 9d77175947e14a7b9aa0944520f30ebd (DEFAULT_TABLE table=test-table [id=41965de5be844a49a46388e3b929dec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:18:57.022491 14358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d77175947e14a7b9aa0944520f30ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:57.024724 14572 tablet_bootstrap.cc:492] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Bootstrap starting.
I20260620 09:18:57.024982 14574 tablet_bootstrap.cc:492] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a: Bootstrap starting.
I20260620 09:18:57.025365 14572 tablet_bootstrap.cc:654] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:57.025578 14574 tablet_bootstrap.cc:654] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:57.025942 14572 log.cc:824] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:57.026070 14574 log.cc:824] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:57.026283 14573 tablet_bootstrap.cc:492] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c: Bootstrap starting.
I20260620 09:18:57.026767 14574 tablet_bootstrap.cc:492] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a: No bootstrap required, opened a new log
I20260620 09:18:57.026839 14574 ts_tablet_manager.cc:1397] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:18:57.026885 14573 tablet_bootstrap.cc:654] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:57.027364 14573 log.cc:824] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:57.027912 14572 tablet_bootstrap.cc:492] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: No bootstrap required, opened a new log
I20260620 09:18:57.027998 14572 ts_tablet_manager.cc:1397] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:18:57.028118 14573 tablet_bootstrap.cc:492] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c: No bootstrap required, opened a new log
I20260620 09:18:57.028185 14573 ts_tablet_manager.cc:1397] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:18:57.028821 14574 raft_consensus.cc:348] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } }
I20260620 09:18:57.029012 14574 raft_consensus.cc:374] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:57.029049 14574 raft_consensus.cc:729] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770d42b044b94d56a1ba645952e4496a, State: Initialized, Role: FOLLOWER
I20260620 09:18:57.029165 14574 consensus_queue.cc:260] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } }
I20260620 09:18:57.029446 14424 heartbeater.cc:503] Master 127.0.98.62:35695 was elected leader, sending a full tablet report...
I20260620 09:18:57.029431 14574 ts_tablet_manager.cc:1428] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:18:57.029938 14572 raft_consensus.cc:348] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } }
I20260620 09:18:57.030062 14572 raft_consensus.cc:374] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:57.030081 14572 raft_consensus.cc:729] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61fc3e9229f41b0b30a1e2e5adebafe, State: Initialized, Role: FOLLOWER
I20260620 09:18:57.030167 14572 consensus_queue.cc:260] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } }
I20260620 09:18:57.030211 14573 raft_consensus.cc:348] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } }
I20260620 09:18:57.030380 14573 raft_consensus.cc:374] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:57.030413 14572 ts_tablet_manager.cc:1428] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 09:18:57.030426 14573 raft_consensus.cc:729] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa09c96387c49359a077279b4a33a5c, State: Initialized, Role: FOLLOWER
I20260620 09:18:57.030529 14290 heartbeater.cc:503] Master 127.0.98.62:35695 was elected leader, sending a full tablet report...
I20260620 09:18:57.030545 14573 consensus_queue.cc:260] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } }
I20260620 09:18:57.030804 14573 ts_tablet_manager.cc:1428] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:18:57.030849 14558 heartbeater.cc:503] Master 127.0.98.62:35695 was elected leader, sending a full tablet report...
W20260620 09:18:57.098544 14425 tablet.cc:2369] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:18:57.201376 14291 tablet.cc:2369] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:18:57.225394 14578 raft_consensus.cc:482] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:18:57.225528 14578 raft_consensus.cc:504] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } }
I20260620 09:18:57.225885 14578 leader_election.cc:288] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293), c61fc3e9229f41b0b30a1e2e5adebafe (127.0.98.1:35131)
I20260620 09:18:57.228715 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.226162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54525 (local address 127.0.98.3:41293)
0620 09:18:57.226224 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.226226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.226236 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:18:57.226266 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.226267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.226278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.226379 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:57.226511 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.227023 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.228014 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.228407 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.228459 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.228468 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.228614 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.228652 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:18:57.228716 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.226251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33153 (local address 127.0.98.1:35131)
0620 09:18:57.226313 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.226315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.226399 (+ 84us) server_negotiation.cc:407] Connection header received
0620 09:18:57.226454 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.226455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.226466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.226516 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:57.226639 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.227137 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.228080 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.228434 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.228497 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.228507 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.228616 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.228651 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:18:57.228724 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.226073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41293 (local address 127.0.98.2:54525)
0620 09:18:57.226121 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.226132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.226182 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.226367 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.226368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.226370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:57.226440 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.226443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.227090 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.227091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.227921 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.227924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.228473 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.228474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.228550 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.228553 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.228598 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.228638 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 09:18:57.228737 14581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.226181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35131 (local address 127.0.98.2:33153)
0620 09:18:57.226337 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.226347 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.226405 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.226523 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.226524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.226525 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:57.226586 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.226587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.227211 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.227212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.228000 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.228002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.228505 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.228506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.228555 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.228558 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.228605 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.228644 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260620 09:18:57.229069 14512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d77175947e14a7b9aa0944520f30ebd" candidate_uuid: "770d42b044b94d56a1ba645952e4496a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa09c96387c49359a077279b4a33a5c" is_pre_election: true
I20260620 09:18:57.229211 14512 raft_consensus.cc:2454] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770d42b044b94d56a1ba645952e4496a in term 0.
I20260620 09:18:57.229440 14312 leader_election.cc:302] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa09c96387c49359a077279b4a33a5c, 770d42b044b94d56a1ba645952e4496a; no voters:
I20260620 09:18:57.229550 14244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d77175947e14a7b9aa0944520f30ebd" candidate_uuid: "770d42b044b94d56a1ba645952e4496a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" is_pre_election: true
I20260620 09:18:57.229597 14578 raft_consensus.cc:2781] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:18:57.229637 14578 raft_consensus.cc:482] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:18:57.229664 14578 raft_consensus.cc:3037] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:57.229684 14244 raft_consensus.cc:2454] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770d42b044b94d56a1ba645952e4496a in term 0.
I20260620 09:18:57.230352 14578 raft_consensus.cc:504] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } }
I20260620 09:18:57.230458 14578 leader_election.cc:288] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [CANDIDATE]: Term 1 election: Requested vote from peers 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293), c61fc3e9229f41b0b30a1e2e5adebafe (127.0.98.1:35131)
I20260620 09:18:57.230623 14244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d77175947e14a7b9aa0944520f30ebd" candidate_uuid: "770d42b044b94d56a1ba645952e4496a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe"
I20260620 09:18:57.230593 14512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d77175947e14a7b9aa0944520f30ebd" candidate_uuid: "770d42b044b94d56a1ba645952e4496a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa09c96387c49359a077279b4a33a5c"
I20260620 09:18:57.230686 14512 raft_consensus.cc:3037] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:57.230686 14244 raft_consensus.cc:3037] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:57.231510 14244 raft_consensus.cc:2454] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770d42b044b94d56a1ba645952e4496a in term 1.
I20260620 09:18:57.231510 14512 raft_consensus.cc:2454] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770d42b044b94d56a1ba645952e4496a in term 1.
I20260620 09:18:57.231659 14312 leader_election.cc:302] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 770d42b044b94d56a1ba645952e4496a, c61fc3e9229f41b0b30a1e2e5adebafe; no voters:
I20260620 09:18:57.231758 14578 raft_consensus.cc:2781] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:18:57.231902 14578 raft_consensus.cc:686] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 1 LEADER]: Becoming Leader. State: Replica: 770d42b044b94d56a1ba645952e4496a, State: Running, Role: LEADER
I20260620 09:18:57.231983 14578 consensus_queue.cc:237] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } }
I20260620 09:18:57.232757 14099 catalog_manager.cc:5495] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a reported cstate change: term changed from 0 to 1, leader changed from <none> to 770d42b044b94d56a1ba645952e4496a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "770d42b044b94d56a1ba645952e4496a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } health_report { overall_health: HEALTHY } } }
W20260620 09:18:57.240921 14559 tablet.cc:2369] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:18:57.244203 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.242262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.0.98.2:38951)
0620 09:18:57.242296 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.242298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.242306 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:18:57.242349 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.242350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.242367 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.242421 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:18:57.242519 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.242968 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.243647 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.243830 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.243876 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.243980 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:18:57.244069 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:18:57.244122 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.244133 (+ 11us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.244149 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 09:18:57.245824 14244 raft_consensus.cc:1261] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Refusing update from remote peer 770d42b044b94d56a1ba645952e4496a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:18:57.245824 14512 raft_consensus.cc:1261] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c [term 1 FOLLOWER]: Refusing update from remote peer 770d42b044b94d56a1ba645952e4496a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:18:57.246137 14578 consensus_queue.cc:1035] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:18:57.246281 14582 consensus_queue.cc:1035] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:18:57.248368 14585 mvcc.cc:205] Tried to move back new op lower bound from 7298855474156486656 to 7298855474102808576. Current Snapshot: MvccSnapshot[applied={T|T < 7298855474156486656}]
I20260620 09:18:57.248531 14586 mvcc.cc:205] Tried to move back new op lower bound from 7298855474156486656 to 7298855474102808576. Current Snapshot: MvccSnapshot[applied={T|T < 7298855474156486656}]
I20260620 09:18:57.249305 14587 mvcc.cc:205] Tried to move back new op lower bound from 7298855474156486656 to 7298855474102808576. Current Snapshot: MvccSnapshot[applied={T|T < 7298855474156486656 or (T in {7298855474156486656})}]
I20260620 09:18:57.250406 14099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:18:57.250484 14099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:18:57.251155 14099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Sending DeleteTablet for 3 replicas of tablet 9d77175947e14a7b9aa0944520f30ebd
I20260620 09:18:57.251458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 14161
I20260620 09:18:57.251466 14224 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d77175947e14a7b9aa0944520f30ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59816
I20260620 09:18:57.251469 14492 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d77175947e14a7b9aa0944520f30ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35786
I20260620 09:18:57.252184 14595 tablet_replica.cc:330] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c: stopping tablet replica
I20260620 09:18:57.252302 14595 raft_consensus.cc:2229] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:18:57.252420 14595 raft_consensus.cc:2258] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:18:57.252873 14358 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d77175947e14a7b9aa0944520f30ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40890
I20260620 09:18:57.252897 14595 ts_tablet_manager.cc:1905] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:18:57.253165 14596 tablet_replica.cc:330] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a: stopping tablet replica
I20260620 09:18:57.253258 14596 raft_consensus.cc:2229] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:18:57.253347 14596 raft_consensus.cc:2258] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:18:57.253720 14596 ts_tablet_manager.cc:1905] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:18:57.254801 14595 ts_tablet_manager.cc:1918] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:18:57.254930 14595 log.cc:1196] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/9d77175947e14a7b9aa0944520f30ebd
I20260620 09:18:57.255131 14596 ts_tablet_manager.cc:1918] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:18:57.255196 14596 log.cc:1196] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/9d77175947e14a7b9aa0944520f30ebd
I20260620 09:18:57.255384 14595 ts_tablet_manager.cc:1939] T 9d77175947e14a7b9aa0944520f30ebd P 6fa09c96387c49359a077279b4a33a5c: Deleting consensus metadata
I20260620 09:18:57.255432 14596 ts_tablet_manager.cc:1939] T 9d77175947e14a7b9aa0944520f30ebd P 770d42b044b94d56a1ba645952e4496a: Deleting consensus metadata
I20260620 09:18:57.255787 14084 catalog_manager.cc:4841] TS 770d42b044b94d56a1ba645952e4496a (127.0.98.2:38951): tablet 9d77175947e14a7b9aa0944520f30ebd (table test-table [id=41965de5be844a49a46388e3b929dec2]) successfully deleted
I20260620 09:18:57.255874 14084 catalog_manager.cc:4841] TS 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293): tablet 9d77175947e14a7b9aa0944520f30ebd (table test-table [id=41965de5be844a49a46388e3b929dec2]) successfully deleted
W20260620 09:18:57.256750 14084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35131 (error 108)
I20260620 09:18:57.257062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 14295
I20260620 09:18:57.257155 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.257045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35131 (local address 127.0.0.1:59832)
0620 09:18:57.257087 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.257124 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 09:18:57.257792 14084 catalog_manager.cc:4584] TS c61fc3e9229f41b0b30a1e2e5adebafe (127.0.98.1:35131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d77175947e14a7b9aa0944520f30ebd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35131: connect: Connection refused (error 111)
I20260620 09:18:57.261679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 14429
I20260620 09:18:57.265305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 14067
I20260620 09:18:57.269632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35695
--webserver_interface=127.0.98.62
--webserver_port=36573
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:57.349545 14598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:57.349740 14598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:57.349761 14598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:57.350863 14598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:18:57.350903 14598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:57.350916 14598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:57.350939 14598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:18:57.350960 14598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:18:57.352438 14598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:57.352665 14598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:57.352867 14598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:57.360777 14607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:57.360849 14598 server_base.cc:1034] running on GCE node
W20260620 09:18:57.360781 14604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:57.360757 14605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:57.361506 14598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:57.361776 14598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:57.362907 14598 hybrid_clock.cc:648] HybridClock initialized: now 1781947137362895 us; error 29 us; skew 500 ppm
Jun 20 09:18:57 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:57.365216 14598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:18:57.365648 14598 webserver.cc:460] Webserver started at http://127.0.98.62:36573/ using document root <none> and password file <none>
I20260620 09:18:57.365854 14598 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:57.365911 14598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:57.367238 14598 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:18:57.367852 14614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:57.368010 14598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:57.368093 14598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "a8c65c5d094b4ae19c76864cbbac8ea9"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "3d29bbee7f8c3c8cdb81407b396f3cbf"
server_key_iv: "5f72aaf9d98e5a2d9dced94635212d71"
server_key_version: "encryptionkey@0"
I20260620 09:18:57.368458 14598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:57.383224 14598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:57.385092 14598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:57.385221 14598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:57.388940 14598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35695
I20260620 09:18:57.388983 14666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35695 every 8 connection(s)
I20260620 09:18:57.389319 14598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:18:57.390784 14667 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:18:57.391417 14667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Bootstrap starting.
I20260620 09:18:57.393911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 14598
I20260620 09:18:57.394156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35131
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35695
--webserver_port=45741
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:18:57.399077 14667 log.cc:824] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:57.400862 14667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:18:57.401095 14667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Bootstrap complete.
I20260620 09:18:57.403115 14667 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260620 09:18:57.403331 14667 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c65c5d094b4ae19c76864cbbac8ea9, State: Initialized, Role: FOLLOWER
I20260620 09:18:57.403486 14667 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260620 09:18:57.403560 14667 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:18:57.403630 14667 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:18:57.403681 14667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:18:57.404662 14667 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260620 09:18:57.404793 14667 leader_election.cc:302] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8c65c5d094b4ae19c76864cbbac8ea9; no voters:
I20260620 09:18:57.404943 14667 leader_election.cc:288] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:18:57.404991 14671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:18:57.405131 14671 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 2 LEADER]: Becoming Leader. State: Replica: a8c65c5d094b4ae19c76864cbbac8ea9, State: Running, Role: LEADER
I20260620 09:18:57.405193 14667 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:18:57.405236 14671 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260620 09:18:57.405480 14671 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8c65c5d094b4ae19c76864cbbac8ea9. Latest consensus state: current_term: 2 leader_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } } }
I20260620 09:18:57.405638 14671 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:57.405583 14672 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } } }
I20260620 09:18:57.405825 14672 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:57.406085 14677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:18:57.406838 14677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:18:57.407058 14677 catalog_manager.cc:1259] Loaded cluster ID: 54d10f55162f4fc196d0ef50037f3c15
I20260620 09:18:57.407235 14677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:18:57.407538 14677 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 09:18:57.412639 14677 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 09:18:57.415002 14677 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:18:57.415184 14677 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Loaded TSK: 0
I20260620 09:18:57.415369 14677 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:18:57.476019 14669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:57.476218 14669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:57.476250 14669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:57.477375 14669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:57.477412 14669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:57.477447 14669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:18:57.478880 14669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45741
--tserver_master_addrs=127.0.98.62:35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:57.479094 14669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:57.479290 14669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:57.483985 14694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:57.484091 14697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:57.484181 14669 server_base.cc:1034] running on GCE node
W20260620 09:18:57.484285 14695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:57.484494 14669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:57.484727 14669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:57.485908 14669 hybrid_clock.cc:648] HybridClock initialized: now 1781947137485894 us; error 25 us; skew 500 ppm
Jun 20 09:18:57 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:57.488579 14669 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:18:57.489030 14669 webserver.cc:460] Webserver started at http://127.0.98.1:45741/ using document root <none> and password file <none>
I20260620 09:18:57.489274 14669 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:57.489336 14669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:57.490619 14669 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:18:57.491218 14704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:57.491402 14669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:18:57.491495 14669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "c61fc3e9229f41b0b30a1e2e5adebafe"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "bc522f3779c136b5fc94421ac0d6c3d1"
server_key_iv: "4ef173d5c55294b062c64ea1a817c78b"
server_key_version: "encryptionkey@0"
I20260620 09:18:57.491971 14669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:57.515377 14669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:57.517933 14669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:57.518090 14669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:57.518365 14669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:57.518922 14711 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:18:57.520385 14669 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:18:57.520440 14669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 09:18:57.520470 14669 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:18:57.521268 14669 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:18:57.521306 14669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 09:18:57.521382 14711 tablet_bootstrap.cc:492] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Bootstrap starting.
I20260620 09:18:57.527807 14669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35131
I20260620 09:18:57.528270 14669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:18:57.528731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 14669
I20260620 09:18:57.529036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38951
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35695
--webserver_port=36293
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:18:57.531498 14711 log.cc:824] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:57.531731 14818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35131 every 8 connection(s)
I20260620 09:18:57.535365 14711 tablet_bootstrap.cc:492] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:18:57.535669 14711 tablet_bootstrap.cc:492] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Bootstrap complete.
I20260620 09:18:57.536731 14711 ts_tablet_manager.cc:1397] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260620 09:18:57.537958 14711 raft_consensus.cc:348] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } }
I20260620 09:18:57.538237 14711 raft_consensus.cc:729] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61fc3e9229f41b0b30a1e2e5adebafe, State: Initialized, Role: FOLLOWER
I20260620 09:18:57.538360 14711 consensus_queue.cc:260] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } }
I20260620 09:18:57.538589 14711 ts_tablet_manager.cc:1428] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
Jun 20 09:18:57 dist-test-slave-rq8c krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:57.543848 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.531050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56549 (local address 127.0.98.62:35695)
0620 09:18:57.532758 (+ 1708us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.532767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.532811 (+ 44us) server_negotiation.cc:407] Connection header received
0620 09:18:57.532932 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.532938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.532989 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.533108 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:57.533972 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.535327 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.537139 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.537728 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.537809 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.540555 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:57.540560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:57.540560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:57.540572 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:57.542262 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:57.542610 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:57.542613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:57.542614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:57.542715 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:57.543007 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:57.543008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:57.543008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:57.543225 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:57.543304 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.543570 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.543712 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1661,"thread_start_us":65,"threads_started":1}
I20260620 09:18:57.543913 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.530981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:56549)
0620 09:18:57.531866 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.531904 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.532464 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.533182 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.533187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.533534 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:57.533808 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.533816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.535641 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.535642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.537040 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.537043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.537947 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.537947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.538080 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.538896 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:57.538906 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:57.540463 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:57.542332 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:57.542337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:57.542339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:57.542537 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:57.542810 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:57.542811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:57.542811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:57.542943 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:57.543280 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:57.543284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:57.543430 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.543524 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.543700 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":99,"threads_started":1}
I20260620 09:18:57.544560 14819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35695
I20260620 09:18:57.544652 14819 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:57.544843 14819 heartbeater.cc:511] Master 127.0.98.62:35695 requested a full tablet report, sending...
I20260620 09:18:57.545553 14631 ts_manager.cc:194] Registered new tserver with Master: c61fc3e9229f41b0b30a1e2e5adebafe (127.0.98.1:35131)
I20260620 09:18:57.546764 14631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56549
I20260620 09:18:57.547564 14819 heartbeater.cc:503] Master 127.0.98.62:35695 was elected leader, sending a full tablet report...
Jun 20 09:18:57 dist-test-slave-rq8c krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:18:57.553750 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.546221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.0.98.1:35131)
0620 09:18:57.546353 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.546357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.546509 (+ 152us) server_negotiation.cc:407] Connection header received
0620 09:18:57.546580 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.546583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.546597 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.546650 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:57.547066 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.548223 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.548905 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.549358 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.549423 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.551358 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:57.551364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:57.551364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:57.551369 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:57.552690 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:57.553083 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:57.553086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:57.553086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:57.553133 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:57.553295 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:57.553296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:57.553296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:57.553410 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:57.553461 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.553621 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.553688 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260620 09:18:57.553793 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.546162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35131 (local address 127.0.0.1:59838)
0620 09:18:57.546418 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.546444 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.546509 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.546658 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.546661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.546946 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:57.546997 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.546999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.548293 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.548293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.548837 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.548839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.549429 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.549431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.549486 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.549902 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:57.549907 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:57.551285 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:57.552760 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:57.552765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:57.552767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:57.553014 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:57.553183 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:57.553184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:57.553184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:57.553245 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:57.553458 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:57.553463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:57.553511 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.553593 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.553700 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":92,"threads_started":1}
I20260620 09:18:57.554335 14739 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d77175947e14a7b9aa0944520f30ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59838
I20260620 09:18:57.554872 14828 tablet_replica.cc:330] stopping tablet replica
I20260620 09:18:57.554972 14828 raft_consensus.cc:2229] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:18:57.555033 14828 raft_consensus.cc:2258] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:18:57.555322 14828 ts_tablet_manager.cc:1905] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:18:57.557456 14828 ts_tablet_manager.cc:1918] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:18:57.557528 14828 log.cc:1196] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal/wals/9d77175947e14a7b9aa0944520f30ebd
I20260620 09:18:57.557732 14828 ts_tablet_manager.cc:1939] T 9d77175947e14a7b9aa0944520f30ebd P c61fc3e9229f41b0b30a1e2e5adebafe: Deleting consensus metadata
I20260620 09:18:57.558225 14616 catalog_manager.cc:4841] TS c61fc3e9229f41b0b30a1e2e5adebafe (127.0.98.1:35131): tablet 9d77175947e14a7b9aa0944520f30ebd (table test-table [id=41965de5be844a49a46388e3b929dec2]) successfully deleted
W20260620 09:18:57.614261 14821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:57.614456 14821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:57.614477 14821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:57.615595 14821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:57.615655 14821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:57.615681 14821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:18:57.617053 14821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36293
--tserver_master_addrs=127.0.98.62:35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:57.617254 14821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:57.617455 14821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:57.621981 14834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:57.621963 14833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:57.622025 14821 server_base.cc:1034] running on GCE node
W20260620 09:18:57.621961 14836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:57.622313 14821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:57.622507 14821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:57.623651 14821 hybrid_clock.cc:648] HybridClock initialized: now 1781947137623633 us; error 30 us; skew 500 ppm
Jun 20 09:18:57 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:57.625871 14821 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:18:57.626266 14821 webserver.cc:460] Webserver started at http://127.0.98.2:36293/ using document root <none> and password file <none>
I20260620 09:18:57.626448 14821 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:57.626505 14821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:57.627794 14821 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:18:57.628310 14843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:57.628458 14821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:57.628533 14821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "770d42b044b94d56a1ba645952e4496a"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "4168d476e84ac55beaf190c63a151ed5"
server_key_iv: "1302614463de437f55a7af17f9cc9cd5"
server_key_version: "encryptionkey@0"
I20260620 09:18:57.628847 14821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:57.636541 14821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:57.638178 14821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:57.638298 14821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:57.638482 14821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:57.638746 14821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:57.638775 14821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:57.638803 14821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:57.638833 14821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:57.644287 14821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38951
I20260620 09:18:57.644349 14956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38951 every 8 connection(s)
I20260620 09:18:57.644655 14821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:18:57 dist-test-slave-rq8c krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:57.653015 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.645262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35035 (local address 127.0.98.62:35695)
0620 09:18:57.645304 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.645306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.645528 (+ 222us) server_negotiation.cc:407] Connection header received
0620 09:18:57.646083 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.646084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.646103 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.646151 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:57.647123 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.647605 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.648589 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.648781 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.648841 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.650709 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:57.650710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:57.650710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:57.650718 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:57.651807 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:57.652159 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:57.652160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:57.652160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:57.652210 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:57.652424 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:57.652425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:57.652425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:57.652529 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:57.652584 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.652834 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.652922 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:18:57.653151 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.645258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:35035)
0620 09:18:57.645412 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.645446 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.645977 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.646231 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.646235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.646790 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:57.647033 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.647039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.647703 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.647703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.648512 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.648514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.648845 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.648846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.648972 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.649389 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:57.649396 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:57.650628 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:57.651888 (+ 1260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:57.651894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:57.651897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:57.652098 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:57.652261 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:57.652262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:57.652262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:57.652367 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:57.652581 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:57.652586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:57.652715 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.652799 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.652946 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260620 09:18:57.653573 14957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35695
I20260620 09:18:57.653674 14957 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:57.653836 14957 heartbeater.cc:511] Master 127.0.98.62:35695 requested a full tablet report, sending...
I20260620 09:18:57.654160 14631 ts_manager.cc:194] Registered new tserver with Master: 770d42b044b94d56a1ba645952e4496a (127.0.98.2:38951)
I20260620 09:18:57.654188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 14821
I20260620 09:18:57.654413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41293
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35695
--webserver_port=35959
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:18:57.654544 14631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35035
W20260620 09:18:57.733707 14960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:57.733906 14960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:57.733928 14960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:57.735060 14960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:57.735100 14960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:57.735126 14960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:18:57.736565 14960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35959
--tserver_master_addrs=127.0.98.62:35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:57.736783 14960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:57.736987 14960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:57.741578 14966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:57.741585 14968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:57.741585 14965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:57.741818 14960 server_base.cc:1034] running on GCE node
I20260620 09:18:57.741988 14960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:57.742182 14960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:57.743325 14960 hybrid_clock.cc:648] HybridClock initialized: now 1781947137743301 us; error 29 us; skew 500 ppm
Jun 20 09:18:57 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:57.748996 14960 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:18:57.749413 14960 webserver.cc:460] Webserver started at http://127.0.98.3:35959/ using document root <none> and password file <none>
I20260620 09:18:57.749611 14960 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:57.749670 14960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:57.750892 14960 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:18:57.751390 14975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:57.751534 14960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:57.751610 14960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "6fa09c96387c49359a077279b4a33a5c"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "a1e329a15927fef8080c816bb6d6e812"
server_key_iv: "43f58a2fc1c7030e95866f7a96767f30"
server_key_version: "encryptionkey@0"
I20260620 09:18:57.751972 14960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:57.768416 14960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:57.770388 14960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:57.770519 14960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:57.770728 14960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:57.771039 14960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:57.771076 14960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:57.771106 14960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:57.771129 14960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:57.776562 14960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41293
I20260620 09:18:57.776613 15088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41293 every 8 connection(s)
I20260620 09:18:57.776937 14960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:18:57.778961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 14960
Jun 20 09:18:57 dist-test-slave-rq8c krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:18:57.785465 15093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.779356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.0.98.62:35695)
0620 09:18:57.779484 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.779486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.779493 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:18:57.779514 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.779514 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.779524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.779579 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:57.780132 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.780604 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.781275 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.781464 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.781515 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.783045 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:57.783046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:57.783046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:57.783054 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:57.784499 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:57.784786 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:57.784787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:57.784788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:57.784842 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:57.784997 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:57.784997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:57.784998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:57.785108 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:57.785251 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.785290 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.785364 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260620 09:18:57.785765 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.777577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38955 (local address 127.0.98.62:35695)
0620 09:18:57.777638 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.777640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.777859 (+ 219us) server_negotiation.cc:407] Connection header received
0620 09:18:57.778340 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.778341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.778351 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.778415 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:57.779512 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.780050 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.781042 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.781238 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.781291 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.783387 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:57.783388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:57.783389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:57.783397 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:57.784588 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:57.784939 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:57.784940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:57.784940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:57.784987 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:57.785195 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:57.785196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:57.785197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:57.785293 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:57.785354 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.785631 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.785692 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:18:57.786051 15091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.777559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:38955)
0620 09:18:57.777717 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.777759 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.778257 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.778478 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.778482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.779158 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:57.779417 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.779426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.780135 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.780136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.780965 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.780968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.781305 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.781306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.781441 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.781895 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:57.781903 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:57.783303 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:57.784652 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:57.784658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:57.784660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:57.784876 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:57.785040 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:57.785040 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:57.785041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:57.785143 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:57.785346 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:57.785352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:57.785507 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.785615 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.785780 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260620 09:18:57.786453 15089 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35695
I20260620 09:18:57.786540 15089 heartbeater.cc:464] Registering TS with master...
I20260620 09:18:57.786708 15089 heartbeater.cc:511] Master 127.0.98.62:35695 requested a full tablet report, sending...
I20260620 09:18:57.787017 14631 ts_manager.cc:194] Registered new tserver with Master: 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293)
I20260620 09:18:57.787352 14631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38955
I20260620 09:18:57.796906 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:18:57.799314 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.797199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.0.98.62:35695)
0620 09:18:57.797254 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.797256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.797379 (+ 123us) server_negotiation.cc:407] Connection header received
0620 09:18:57.797428 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.797429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.797462 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.797518 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:18:57.797608 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.798074 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.798760 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.798936 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.798978 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.799077 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:18:57.799167 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:18:57.799219 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.799234 (+ 15us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.799247 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:18:57.799588 14631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:18:57.801162 14631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:18:57.803289 14739 tablet_service.cc:1467] Processing CreateTablet for tablet 053c070ae0554db6aac14862714cb972 (DEFAULT_TABLE table=test-table [id=b31e49bba62c4b8f8c12cef4dded2803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:18:57.803465 14739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 053c070ae0554db6aac14862714cb972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:57.805603 14711 tablet_bootstrap.cc:492] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe: Bootstrap starting.
I20260620 09:18:57.806018 14711 tablet_bootstrap.cc:654] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:57.806727 15096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.802927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.0.98.2:38951)
0620 09:18:57.803085 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.803090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.803100 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:18:57.803138 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.803141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.803156 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.803228 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:57.803353 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.804383 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.805578 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.806391 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.806458 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.806517 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.806634 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.806664 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260620 09:18:57.806730 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.802808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38951 (local address 127.0.0.1:40914)
0620 09:18:57.802848 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.802859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.802901 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.803235 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.803236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.803237 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:57.803280 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.803282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.804491 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.804492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.805505 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.805507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.806467 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.806468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.806544 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.806565 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.806623 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.806664 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 09:18:57.807166 14711 tablet_bootstrap.cc:492] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe: No bootstrap required, opened a new log
I20260620 09:18:57.807214 14711 ts_tablet_manager.cc:1397] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260620 09:18:57.807216 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.802888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41293 (local address 127.0.0.1:35792)
0620 09:18:57.803049 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.803057 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.803092 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.803961 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.803962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.803964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:57.804002 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.804004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.805101 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.805102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.806131 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.806134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.807000 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.807001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.807061 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.807065 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.807114 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.807151 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260620 09:18:57.807236 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.803151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.0.98.3:41293)
0620 09:18:57.803743 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.803747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.803758 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:18:57.803792 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.803794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.803808 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.803885 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:57.804059 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.804958 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.806196 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.806912 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.806970 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.807023 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.807125 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.807162 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":69,"threads_started":1}
I20260620 09:18:57.807431 14891 tablet_service.cc:1467] Processing CreateTablet for tablet 053c070ae0554db6aac14862714cb972 (DEFAULT_TABLE table=test-table [id=b31e49bba62c4b8f8c12cef4dded2803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:18:57.807519 14711 raft_consensus.cc:348] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:18:57.807574 14711 raft_consensus.cc:374] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:57.807598 14711 raft_consensus.cc:729] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61fc3e9229f41b0b30a1e2e5adebafe, State: Initialized, Role: FOLLOWER
I20260620 09:18:57.807660 14711 consensus_queue.cc:260] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:18:57.807729 14891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 053c070ae0554db6aac14862714cb972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:57.807752 14711 ts_tablet_manager.cc:1428] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:18:57.808413 15023 tablet_service.cc:1467] Processing CreateTablet for tablet 053c070ae0554db6aac14862714cb972 (DEFAULT_TABLE table=test-table [id=b31e49bba62c4b8f8c12cef4dded2803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:18:57.808676 15023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 053c070ae0554db6aac14862714cb972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:18:57.811844 15100 tablet_bootstrap.cc:492] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c: Bootstrap starting.
I20260620 09:18:57.811872 15099 tablet_bootstrap.cc:492] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a: Bootstrap starting.
I20260620 09:18:57.812443 15100 tablet_bootstrap.cc:654] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:57.812577 15099 tablet_bootstrap.cc:654] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a: Neither blocks nor log segments found. Creating new log.
I20260620 09:18:57.812880 15100 log.cc:824] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:57.813026 15099 log.cc:824] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:57.813596 15100 tablet_bootstrap.cc:492] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c: No bootstrap required, opened a new log
I20260620 09:18:57.813669 15099 tablet_bootstrap.cc:492] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a: No bootstrap required, opened a new log
I20260620 09:18:57.813668 15100 ts_tablet_manager.cc:1397] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:18:57.813725 15099 ts_tablet_manager.cc:1397] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:18:57.815531 15099 raft_consensus.cc:348] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:18:57.815531 15100 raft_consensus.cc:348] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:18:57.815732 15099 raft_consensus.cc:374] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:57.815732 15100 raft_consensus.cc:374] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:18:57.815775 15100 raft_consensus.cc:729] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa09c96387c49359a077279b4a33a5c, State: Initialized, Role: FOLLOWER
I20260620 09:18:57.815775 15099 raft_consensus.cc:729] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770d42b044b94d56a1ba645952e4496a, State: Initialized, Role: FOLLOWER
I20260620 09:18:57.815901 15099 consensus_queue.cc:260] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:18:57.815901 15100 consensus_queue.cc:260] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:18:57.816156 15100 ts_tablet_manager.cc:1428] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:18:57.816156 15099 ts_tablet_manager.cc:1428] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:18:57.816179 15089 heartbeater.cc:503] Master 127.0.98.62:35695 was elected leader, sending a full tablet report...
I20260620 09:18:57.816278 14957 heartbeater.cc:503] Master 127.0.98.62:35695 was elected leader, sending a full tablet report...
I20260620 09:18:57.838574 15104 raft_consensus.cc:482] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:18:57.838691 15104 raft_consensus.cc:504] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:18:57.839043 15104 leader_election.cc:288] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293), c61fc3e9229f41b0b30a1e2e5adebafe (127.0.98.1:35131)
I20260620 09:18:57.842015 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.839239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41293 (local address 127.0.98.2:42413)
0620 09:18:57.839295 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.839307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.839364 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.839525 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.839526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.839527 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:57.839594 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.839597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.840284 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.840285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.841110 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.841112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.841748 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.841749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.841840 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.841844 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.841901 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.841941 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 09:18:57.842013 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.839346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42413 (local address 127.0.98.3:41293)
0620 09:18:57.839393 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.839395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.839405 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:18:57.839435 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.839436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.839448 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.839509 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:57.839670 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.840211 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.841220 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.841667 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.841726 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.841735 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.841912 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.841944 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:18:57.842299 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.839422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42149 (local address 127.0.98.1:35131)
0620 09:18:57.839475 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.839478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.839590 (+ 112us) server_negotiation.cc:407] Connection header received
0620 09:18:57.839647 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.839649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.839674 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.839730 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:18:57.839875 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.840452 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.841443 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.841924 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.841980 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.842072 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.842172 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.842224 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:18:57.842298 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.839354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35131 (local address 127.0.98.2:42149)
0620 09:18:57.839516 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.839527 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:57.839587 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:57.839744 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:57.839745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:57.839747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:18:57.839812 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.839814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.840540 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.840541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.841350 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:57.841353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.841998 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.841999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:57.842068 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.842073 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:18:57.842155 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:18:57.842223 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260620 09:18:57.842386 15043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "053c070ae0554db6aac14862714cb972" candidate_uuid: "770d42b044b94d56a1ba645952e4496a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa09c96387c49359a077279b4a33a5c" is_pre_election: true
I20260620 09:18:57.842564 15043 raft_consensus.cc:2454] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770d42b044b94d56a1ba645952e4496a in term 0.
I20260620 09:18:57.842561 14773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "053c070ae0554db6aac14862714cb972" candidate_uuid: "770d42b044b94d56a1ba645952e4496a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" is_pre_election: true
I20260620 09:18:57.842730 14773 raft_consensus.cc:2454] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770d42b044b94d56a1ba645952e4496a in term 0.
I20260620 09:18:57.842783 14845 leader_election.cc:302] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa09c96387c49359a077279b4a33a5c, 770d42b044b94d56a1ba645952e4496a; no voters:
I20260620 09:18:57.842929 15104 raft_consensus.cc:2781] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:18:57.842988 15104 raft_consensus.cc:482] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:18:57.843009 15104 raft_consensus.cc:3037] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:57.843722 15104 raft_consensus.cc:504] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:18:57.843837 15104 leader_election.cc:288] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [CANDIDATE]: Term 1 election: Requested vote from peers 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293), c61fc3e9229f41b0b30a1e2e5adebafe (127.0.98.1:35131)
I20260620 09:18:57.844022 15043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "053c070ae0554db6aac14862714cb972" candidate_uuid: "770d42b044b94d56a1ba645952e4496a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa09c96387c49359a077279b4a33a5c"
I20260620 09:18:57.844045 14773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "053c070ae0554db6aac14862714cb972" candidate_uuid: "770d42b044b94d56a1ba645952e4496a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe"
I20260620 09:18:57.844112 15043 raft_consensus.cc:3037] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:57.844115 14773 raft_consensus.cc:3037] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:18:57.844772 14773 raft_consensus.cc:2454] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770d42b044b94d56a1ba645952e4496a in term 1.
I20260620 09:18:57.844806 15043 raft_consensus.cc:2454] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770d42b044b94d56a1ba645952e4496a in term 1.
I20260620 09:18:57.844934 14845 leader_election.cc:302] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa09c96387c49359a077279b4a33a5c, 770d42b044b94d56a1ba645952e4496a; no voters:
I20260620 09:18:57.845027 15104 raft_consensus.cc:2781] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:18:57.845191 15104 raft_consensus.cc:686] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 1 LEADER]: Becoming Leader. State: Replica: 770d42b044b94d56a1ba645952e4496a, State: Running, Role: LEADER
I20260620 09:18:57.845268 15104 consensus_queue.cc:237] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:18:57.846021 14631 catalog_manager.cc:5495] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a reported cstate change: term changed from 0 to 1, leader changed from <none> to 770d42b044b94d56a1ba645952e4496a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "770d42b044b94d56a1ba645952e4496a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } health_report { overall_health: UNKNOWN } } }
I20260620 09:18:57.851192 15096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.849340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.0.98.2:38951)
0620 09:18:57.849379 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:57.849381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:57.849390 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:18:57.849447 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:57.849448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:57.849463 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:57.849511 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:18:57.849590 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.850028 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.850682 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:57.850858 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:57.850898 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:57.850984 (+ 86us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:18:57.851068 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:18:57.851119 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:57.851131 (+ 12us) server_negotiation.cc:299] Negotiation successful
0620 09:18:57.851145 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:18:57.852707 14773 raft_consensus.cc:1261] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Refusing update from remote peer 770d42b044b94d56a1ba645952e4496a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:18:57.852730 15043 raft_consensus.cc:1261] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c [term 1 FOLLOWER]: Refusing update from remote peer 770d42b044b94d56a1ba645952e4496a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:18:57.852964 15104 consensus_queue.cc:1035] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:18:57.853080 15106 consensus_queue.cc:1035] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:18:57.854957 15108 mvcc.cc:205] Tried to move back new op lower bound from 7298855476642664448 to 7298855476614762496. Current Snapshot: MvccSnapshot[applied={T|T < 7298855476642664448}]
I20260620 09:18:57.855669 15110 mvcc.cc:205] Tried to move back new op lower bound from 7298855476642664448 to 7298855476614762496. Current Snapshot: MvccSnapshot[applied={T|T < 7298855476642664448}]
I20260620 09:18:57.855674 15109 mvcc.cc:205] Tried to move back new op lower bound from 7298855476642664448 to 7298855476614762496. Current Snapshot: MvccSnapshot[applied={T|T < 7298855476642664448}]
I20260620 09:18:57.857257 14631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:18:57.857334 14631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:18:57.857945 14631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Sending DeleteTablet for 3 replicas of tablet 053c070ae0554db6aac14862714cb972
I20260620 09:18:57.858196 14891 tablet_service.cc:1514] Processing DeleteTablet for tablet 053c070ae0554db6aac14862714cb972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40914
I20260620 09:18:57.858244 15023 tablet_service.cc:1514] Processing DeleteTablet for tablet 053c070ae0554db6aac14862714cb972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35792
I20260620 09:18:57.858275 14739 tablet_service.cc:1514] Processing DeleteTablet for tablet 053c070ae0554db6aac14862714cb972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:18:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59838
I20260620 09:18:57.858364 14828 tablet_replica.cc:330] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe: stopping tablet replica
I20260620 09:18:57.858417 14828 raft_consensus.cc:2229] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:18:57.858469 14828 raft_consensus.cc:2258] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:18:57.858484 15119 tablet_replica.cc:330] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c: stopping tablet replica
I20260620 09:18:57.858486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 14669
I20260620 09:18:57.859000 15119 raft_consensus.cc:2229] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:18:57.859102 15119 raft_consensus.cc:2258] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:18:57.859225 15118 tablet_replica.cc:330] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a: stopping tablet replica
I20260620 09:18:57.859308 15118 raft_consensus.cc:2229] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:18:57.859376 15118 raft_consensus.cc:2258] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:18:57.859750 15119 ts_tablet_manager.cc:1905] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:18:57.861037 15118 ts_tablet_manager.cc:1905] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:18:57.862038 15118 ts_tablet_manager.cc:1918] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:18:57.862079 15118 log.cc:1196] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/053c070ae0554db6aac14862714cb972
I20260620 09:18:57.862244 15118 ts_tablet_manager.cc:1939] T 053c070ae0554db6aac14862714cb972 P 770d42b044b94d56a1ba645952e4496a: Deleting consensus metadata
I20260620 09:18:57.862528 14616 catalog_manager.cc:4841] TS 770d42b044b94d56a1ba645952e4496a (127.0.98.2:38951): tablet 053c070ae0554db6aac14862714cb972 (table test-table [id=b31e49bba62c4b8f8c12cef4dded2803]) successfully deleted
W20260620 09:18:57.863088 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35131 (error 108)
I20260620 09:18:57.863319 15119 ts_tablet_manager.cc:1918] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:18:57.863348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 14821
I20260620 09:18:57.863420 15119 log.cc:1196] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/053c070ae0554db6aac14862714cb972
I20260620 09:18:57.864078 15119 ts_tablet_manager.cc:1939] T 053c070ae0554db6aac14862714cb972 P 6fa09c96387c49359a077279b4a33a5c: Deleting consensus metadata
I20260620 09:18:57.864161 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:57.863461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35131 (local address 127.0.0.1:59848)
0620 09:18:57.864087 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:57.864130 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603}
W20260620 09:18:57.865056 14616 catalog_manager.cc:4584] TS c61fc3e9229f41b0b30a1e2e5adebafe (127.0.98.1:35131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 053c070ae0554db6aac14862714cb972: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35131: connect: Connection refused (error 111)
I20260620 09:18:57.865414 14616 catalog_manager.cc:4841] TS 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293): tablet 053c070ae0554db6aac14862714cb972 (table test-table [id=b31e49bba62c4b8f8c12cef4dded2803]) successfully deleted
I20260620 09:18:57.868275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 14960
I20260620 09:18:57.871963 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 14598
I20260620 09:18:57.876276 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35695
--webserver_interface=127.0.98.62
--webserver_port=36573
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:57.953804 15121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:57.954003 15121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:57.954025 15121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:57.955104 15121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:18:57.955137 15121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:57.955148 15121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:57.955160 15121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:18:57.955170 15121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:18:57.956564 15121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:57.956741 15121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:57.956907 15121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:57.961596 15129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:57.961615 15127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:57.961660 15121 server_base.cc:1034] running on GCE node
W20260620 09:18:57.961611 15126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:57.962076 15121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:57.962344 15121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:57.963485 15121 hybrid_clock.cc:648] HybridClock initialized: now 1781947137963470 us; error 27 us; skew 500 ppm
Jun 20 09:18:57 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:57.965819 15121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:18:57.966209 15121 webserver.cc:460] Webserver started at http://127.0.98.62:36573/ using document root <none> and password file <none>
I20260620 09:18:57.966395 15121 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:57.966441 15121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:57.967741 15121 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:18:57.968322 15136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:57.968459 15121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:18:57.968535 15121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "a8c65c5d094b4ae19c76864cbbac8ea9"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "3d29bbee7f8c3c8cdb81407b396f3cbf"
server_key_iv: "5f72aaf9d98e5a2d9dced94635212d71"
server_key_version: "encryptionkey@0"
I20260620 09:18:57.968868 15121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:58.008361 15121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:58.010488 15121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:58.010625 15121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:58.014199 15121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35695
I20260620 09:18:58.014256 15188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35695 every 8 connection(s)
I20260620 09:18:58.014595 15121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:18:58.016127 15189 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:18:58.016760 15189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Bootstrap starting.
I20260620 09:18:58.020629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 15121
I20260620 09:18:58.020949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35131
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35695
--webserver_port=45741
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:18:58.024956 15189 log.cc:824] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:58.027120 15189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:18:58.027388 15189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9: Bootstrap complete.
I20260620 09:18:58.029481 15189 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260620 09:18:58.029706 15189 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c65c5d094b4ae19c76864cbbac8ea9, State: Initialized, Role: FOLLOWER
I20260620 09:18:58.029842 15189 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260620 09:18:58.029910 15189 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:18:58.029958 15189 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:18:58.030006 15189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:18:58.030943 15189 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260620 09:18:58.031028 15189 leader_election.cc:302] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8c65c5d094b4ae19c76864cbbac8ea9; no voters:
I20260620 09:18:58.031176 15189 leader_election.cc:288] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 09:18:58.031222 15193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 09:18:58.031338 15193 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [term 3 LEADER]: Becoming Leader. State: Replica: a8c65c5d094b4ae19c76864cbbac8ea9, State: Running, Role: LEADER
I20260620 09:18:58.031432 15189 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:18:58.031442 15193 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } }
I20260620 09:18:58.031782 15193 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8c65c5d094b4ae19c76864cbbac8ea9. Latest consensus state: current_term: 3 leader_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } } }
I20260620 09:18:58.031867 15193 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:58.032020 15195 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c65c5d094b4ae19c76864cbbac8ea9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35695 } } }
I20260620 09:18:58.032076 15195 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c65c5d094b4ae19c76864cbbac8ea9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:18:58.032296 15200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:18:58.033038 15200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:18:58.033133 15200 catalog_manager.cc:1259] Loaded cluster ID: 54d10f55162f4fc196d0ef50037f3c15
I20260620 09:18:58.033165 15200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 09:18:58.033361 15200 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9301a5e9ad google::LogMessage::Fail() at ??:0
@ 0x7f9301a6188a google::LogMessage::SendToLog() at ??:0
@ 0x7f9301a5e4de google::LogMessage::Flush() at ??:0
@ 0x7f9301a5fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f930997f0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f93099888b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f930392c0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9303921954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f93047856db start_thread at ??:0
@ 0x7f930055c61f clone at ??:0
W20260620 09:18:58.104380 15191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:58.104571 15191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:58.104600 15191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:58.105818 15191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:58.105857 15191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:58.105893 15191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:18:58.107319 15191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45741
--tserver_master_addrs=127.0.98.62:35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:58.107542 15191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:58.107764 15191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:58.113391 15228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:58.113454 15226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:58.113519 15225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:58.113643 15191 server_base.cc:1034] running on GCE node
I20260620 09:18:58.113864 15191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:58.114116 15191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:58.115271 15191 hybrid_clock.cc:648] HybridClock initialized: now 1781947138115258 us; error 26 us; skew 500 ppm
Jun 20 09:18:58 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:58.117738 15191 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:18:58.118132 15191 webserver.cc:460] Webserver started at http://127.0.98.1:45741/ using document root <none> and password file <none>
I20260620 09:18:58.118335 15191 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:58.118393 15191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:58.119678 15191 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:18:58.120303 15235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:58.120453 15191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:18:58.120524 15191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "c61fc3e9229f41b0b30a1e2e5adebafe"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "bc522f3779c136b5fc94421ac0d6c3d1"
server_key_iv: "4ef173d5c55294b062c64ea1a817c78b"
server_key_version: "encryptionkey@0"
I20260620 09:18:58.120841 15191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:58.144538 15191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:58.146678 15191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:58.146809 15191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:58.147027 15191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:58.147526 15242 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:18:58.149458 15191 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:18:58.149506 15191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 09:18:58.149539 15191 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:18:58.150153 15191 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:18:58.150184 15191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 09:18:58.150274 15242 tablet_bootstrap.cc:492] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe: Bootstrap starting.
I20260620 09:18:58.156952 15191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35131
I20260620 09:18:58.157130 15349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35131 every 8 connection(s)
I20260620 09:18:58.157372 15191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:18:58.161010 15242 log.cc:824] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe: Log is configured to *not* fsync() on all Append() calls
I20260620 09:18:58.163560 15242 tablet_bootstrap.cc:492] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:18:58.163868 15242 tablet_bootstrap.cc:492] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe: Bootstrap complete.
I20260620 09:18:58.164639 15242 ts_tablet_manager.cc:1397] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260620 09:18:58.165889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 15191
I20260620 09:18:58.166115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38951
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35695
--webserver_port=36293
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:18:58.165982 15242 raft_consensus.cc:348] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:18:58.166325 15242 raft_consensus.cc:729] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61fc3e9229f41b0b30a1e2e5adebafe, State: Initialized, Role: FOLLOWER
I20260620 09:18:58.166476 15242 consensus_queue.cc:260] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:18:58.166744 15242 ts_tablet_manager.cc:1428] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260620 09:18:58.221647 15352 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:18:58.158014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:34263)
0620 09:18:58.158191 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:58.158224 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:58.158664 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:58.221475 (+ 62811us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
W20260620 09:18:58.221895 15350 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35695 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35695: Client connection negotiation failed: client connection to 127.0.98.62:35695: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:18:58.222239 15352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:58.222121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:47007)
0620 09:18:58.222176 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:58.222214 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 09:18:58.222509 15352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:58.222435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:44443)
0620 09:18:58.222471 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:58.222492 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 09:18:58.222625 15350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:18:58.245860 15355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:58.246057 15355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:58.246079 15355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:58.247202 15355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:58.247243 15355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:58.247272 15355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:18:58.248744 15355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36293
--tserver_master_addrs=127.0.98.62:35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:58.248953 15355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:58.249156 15355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:58.253772 15360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:58.253873 15355 server_base.cc:1034] running on GCE node
W20260620 09:18:58.253772 15363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:58.253983 15361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:58.254179 15355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:58.254385 15355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:58.255508 15355 hybrid_clock.cc:648] HybridClock initialized: now 1781947138255500 us; error 26 us; skew 500 ppm
Jun 20 09:18:58 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:58.257797 15355 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:18:58.258168 15355 webserver.cc:460] Webserver started at http://127.0.98.2:36293/ using document root <none> and password file <none>
I20260620 09:18:58.258343 15355 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:58.258383 15355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:58.259528 15355 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:18:58.260066 15370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:58.260200 15355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:58.260290 15355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "770d42b044b94d56a1ba645952e4496a"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "4168d476e84ac55beaf190c63a151ed5"
server_key_iv: "1302614463de437f55a7af17f9cc9cd5"
server_key_version: "encryptionkey@0"
I20260620 09:18:58.260625 15355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:58.278752 15355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:58.280521 15355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:58.280623 15355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:58.280794 15355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:58.281062 15355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:58.281093 15355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:58.281114 15355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:58.281137 15355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:58.286641 15355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38951
I20260620 09:18:58.286690 15483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38951 every 8 connection(s)
I20260620 09:18:58.286993 15355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:18:58.287945 15486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:58.287572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:38045)
0620 09:18:58.287714 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:58.287787 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
W20260620 09:18:58.288271 15484 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35695 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35695: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
I20260620 09:18:58.288503 15486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:58.288423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:59879)
0620 09:18:58.288461 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:58.288485 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 09:18:58.288722 15486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:58.288659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:55145)
0620 09:18:58.288690 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:58.288708 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 09:18:58.288812 15484 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:18:58.290966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 15355
I20260620 09:18:58.291178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41293
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35695
--webserver_port=35959
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:18:58.367926 15487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:18:58.368108 15487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:18:58.368127 15487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:18:58.369215 15487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:18:58.369252 15487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:18:58.369280 15487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:18:58.370661 15487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35959
--tserver_master_addrs=127.0.98.62:35695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:18:58.370868 15487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:18:58.371069 15487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:18:58.375487 15495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:58.375658 15493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:18:58.375670 15492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:18:58.375819 15487 server_base.cc:1034] running on GCE node
I20260620 09:18:58.375994 15487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:18:58.376202 15487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:18:58.377336 15487 hybrid_clock.cc:648] HybridClock initialized: now 1781947138377308 us; error 37 us; skew 500 ppm
Jun 20 09:18:58 dist-test-slave-rq8c krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:18:58.379556 15487 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:18:58.379987 15487 webserver.cc:460] Webserver started at http://127.0.98.3:35959/ using document root <none> and password file <none>
I20260620 09:18:58.380183 15487 fs_manager.cc:359] Metadata directory not provided
I20260620 09:18:58.380230 15487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:18:58.381412 15487 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:18:58.381914 15502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:58.382079 15487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:58.382157 15487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "6fa09c96387c49359a077279b4a33a5c"
format_stamp: "Formatted at 2026-06-20 09:18:56 on dist-test-slave-rq8c"
server_key: "a1e329a15927fef8080c816bb6d6e812"
server_key_iv: "43f58a2fc1c7030e95866f7a96767f30"
server_key_version: "encryptionkey@0"
I20260620 09:18:58.382459 15487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:18:58.397961 15487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:18:58.400075 15487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:18:58.400213 15487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:18:58.400419 15487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:18:58.400703 15487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:18:58.400734 15487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:58.400764 15487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:18:58.400789 15487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:18:58.406211 15487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41293
I20260620 09:18:58.406280 15615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41293 every 8 connection(s)
I20260620 09:18:58.406572 15487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:18:58.407481 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:58.407140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:34573)
0620 09:18:58.407281 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:58.407348 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
W20260620 09:18:58.407858 15616 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35695 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35695: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
I20260620 09:18:58.408154 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:58.408068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:55405)
0620 09:18:58.408111 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:58.408137 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 09:18:58.408438 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:58.408369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:42763)
0620 09:18:58.408402 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:58.408423 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 09:18:58.408520 15616 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:18:58.415311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 15487
I20260620 09:18:59.223171 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:59.222956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:40115)
0620 09:18:59.223083 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:59.223115 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260620 09:18:59.289315 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:59.289110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:57281)
0620 09:18:59.289222 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:59.289258 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260620 09:18:59.408978 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:59.408803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:36909)
0620 09:18:59.408902 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:59.408934 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260620 09:18:59.481262 15623 raft_consensus.cc:482] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:18:59.481380 15623 raft_consensus.cc:504] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:18:59.481698 15623 leader_election.cc:288] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 770d42b044b94d56a1ba645952e4496a (127.0.98.2:38951), 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293)
Jun 20 09:18:59 dist-test-slave-rq8c krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 09:18:59 dist-test-slave-rq8c krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 09:18:59.491214 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:59.481936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41473 (local address 127.0.98.2:38951)
0620 09:18:59.482093 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:59.482107 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:59.482140 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:18:59.482591 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:59.482593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:59.482633 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:59.482767 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:59.483459 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:59.484338 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:59.485336 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:59.485779 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:59.485873 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:59.487986 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:59.487990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:59.487991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:59.488001 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:59.489615 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:59.489987 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:59.489990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:59.489990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:59.490096 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:59.490355 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:59.490356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:59.490356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:59.490638 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:59.490694 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:59.490993 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 09:18:59.491112 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260620 09:18:59.491215 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:59.482046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42381 (local address 127.0.98.3:41293)
0620 09:18:59.482196 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:18:59.482205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:18:59.482229 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:18:59.482644 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:18:59.482646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:18:59.482680 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:18:59.482793 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:18:59.483460 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:59.484346 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:59.485319 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:18:59.485776 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:18:59.485869 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:59.488364 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:18:59.488368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:18:59.488369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:18:59.488379 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:18:59.489971 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:59.490266 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:59.490269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:59.490269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:59.490365 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:18:59.490529 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:18:59.490530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:18:59.490530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:18:59.490768 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:18:59.490843 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:18:59.490982 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 09:18:59.491092 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260620 09:18:59.491219 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:59.481832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38951 (local address 127.0.98.1:41473)
0620 09:18:59.481874 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:59.481883 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:59.481937 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:59.482805 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:59.482809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:59.483164 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:59.483354 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:59.483360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:59.484425 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:59.484426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:59.485258 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:59.485261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:59.485855 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:59.485856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:59.485965 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:59.486467 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:59.486477 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:59.487891 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:59.489710 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:59.489716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:59.489718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:59.489918 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:59.490163 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:59.490164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:59.490164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:59.490293 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:59.490696 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:59.490702 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:59.490832 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 09:18:59.490930 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 09:18:59.491103 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":53}
I20260620 09:18:59.491312 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:18:59.481940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41293 (local address 127.0.98.1:42381)
0620 09:18:59.482044 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 09:18:59.482055 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:18:59.482120 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:18:59.482812 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:18:59.482813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:18:59.483164 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:18:59.483354 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:59.483360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:59.484432 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:59.484433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:59.485243 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:18:59.485245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:18:59.485984 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:18:59.485985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:18:59.486026 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:18:59.486586 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:18:59.486590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:18:59.488256 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:18:59.490029 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:59.490029 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:59.490029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:59.490196 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:59.490422 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:18:59.490422 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:18:59.490423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:18:59.490474 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:18:59.490830 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:18:59.490831 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:18:59.490874 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 09:18:59.490931 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 09:18:59.491173 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":159,"thread_start_us":56,"threads_started":1}
W20260620 09:18:59.492146 15237 leader_election.cc:341] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 770d42b044b94d56a1ba645952e4496a (127.0.98.2:38951): Not found: Tablet not found: 053c070ae0554db6aac14862714cb972
W20260620 09:18:59.492211 15237 leader_election.cc:341] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293): Not found: Tablet not found: 053c070ae0554db6aac14862714cb972
I20260620 09:18:59.492236 15237 leader_election.cc:302] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c61fc3e9229f41b0b30a1e2e5adebafe; no voters: 6fa09c96387c49359a077279b4a33a5c, 770d42b044b94d56a1ba645952e4496a
I20260620 09:18:59.492331 15623 raft_consensus.cc:2726] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:19:00.223768 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:00.223544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:42015)
0620 09:19:00.223682 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:00.223719 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260620 09:19:00.289944 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:00.289742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:60539)
0620 09:19:00.289865 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:00.289904 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260620 09:19:00.409479 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:00.409310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:35513)
0620 09:19:00.409412 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:00.409444 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
I20260620 09:19:01.169309 15630 raft_consensus.cc:482] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:19:01.169406 15630 raft_consensus.cc:504] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:19:01.169564 15630 leader_election.cc:288] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 770d42b044b94d56a1ba645952e4496a (127.0.98.2:38951), 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293)
W20260620 09:19:01.169942 15237 leader_election.cc:341] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 770d42b044b94d56a1ba645952e4496a (127.0.98.2:38951): Not found: Tablet not found: 053c070ae0554db6aac14862714cb972
W20260620 09:19:01.170013 15237 leader_election.cc:341] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293): Not found: Tablet not found: 053c070ae0554db6aac14862714cb972
I20260620 09:19:01.170053 15237 leader_election.cc:302] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c61fc3e9229f41b0b30a1e2e5adebafe; no voters: 6fa09c96387c49359a077279b4a33a5c, 770d42b044b94d56a1ba645952e4496a
I20260620 09:19:01.170111 15630 raft_consensus.cc:2726] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:19:01.224457 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:01.224218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:38197)
0620 09:19:01.224355 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:01.224405 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260620 09:19:01.290546 15632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:01.290333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:44115)
0620 09:19:01.290467 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:01.290503 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260620 09:19:01.410154 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:01.409910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:47933)
0620 09:19:01.410077 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:01.410117 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20260620 09:19:02.225118 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:02.224901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:57683)
0620 09:19:02.225025 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:02.225068 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260620 09:19:02.291195 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:02.290997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:46495)
0620 09:19:02.291115 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:02.291154 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260620 09:19:02.410734 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:02.410538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:47911)
0620 09:19:02.410654 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:02.410682 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260620 09:19:03.225843 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:03.225601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:46239)
0620 09:19:03.225737 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:03.225790 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
I20260620 09:19:03.291793 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:03.291557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:46901)
0620 09:19:03.291709 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:03.291749 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260620 09:19:03.349620 15639 raft_consensus.cc:482] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:19:03.349716 15639 raft_consensus.cc:504] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:19:03.349841 15639 leader_election.cc:288] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 770d42b044b94d56a1ba645952e4496a (127.0.98.2:38951), 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293)
W20260620 09:19:03.350219 15237 leader_election.cc:341] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 770d42b044b94d56a1ba645952e4496a (127.0.98.2:38951): Not found: Tablet not found: 053c070ae0554db6aac14862714cb972
W20260620 09:19:03.350291 15237 leader_election.cc:341] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293): Not found: Tablet not found: 053c070ae0554db6aac14862714cb972
I20260620 09:19:03.350306 15237 leader_election.cc:302] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c61fc3e9229f41b0b30a1e2e5adebafe; no voters: 6fa09c96387c49359a077279b4a33a5c, 770d42b044b94d56a1ba645952e4496a
I20260620 09:19:03.350378 15639 raft_consensus.cc:2726] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:19:03.411334 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:03.411122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:59773)
0620 09:19:03.411246 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:03.411288 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260620 09:19:04.226522 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:04.226287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:59117)
0620 09:19:04.226432 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:04.226473 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260620 09:19:04.292531 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:04.292256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:44693)
0620 09:19:04.292427 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:04.292474 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260620 09:19:04.411980 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:04.411764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:47233)
0620 09:19:04.411903 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:04.411934 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260620 09:19:05.227146 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:05.226912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:37107)
0620 09:19:05.227057 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:05.227099 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260620 09:19:05.293143 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:05.292938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:49487)
0620 09:19:05.293062 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:05.293099 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260620 09:19:05.412662 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:05.412417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:36133)
0620 09:19:05.412550 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:05.412602 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260620 09:19:06.227929 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:06.227680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:52047)
0620 09:19:06.227837 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:06.227880 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260620 09:19:06.293761 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:06.293544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:47523)
0620 09:19:06.293680 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:06.293714 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260620 09:19:06.413403 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:06.413153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:35045)
0620 09:19:06.413298 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:06.413349 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":98,"threads_started":1}
I20260620 09:19:07.228693 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:07.228403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:35191)
0620 09:19:07.228588 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:07.228633 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260620 09:19:07.294473 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:07.294236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:40217)
0620 09:19:07.294380 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:07.294426 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260620 09:19:07.414093 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:07.413855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:33967)
0620 09:19:07.414008 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:07.414049 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260620 09:19:07.640051 15653 raft_consensus.cc:482] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:19:07.640153 15653 raft_consensus.cc:504] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d42b044b94d56a1ba645952e4496a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38951 } } peers { permanent_uuid: "6fa09c96387c49359a077279b4a33a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41293 } } peers { permanent_uuid: "c61fc3e9229f41b0b30a1e2e5adebafe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } }
I20260620 09:19:07.640287 15653 leader_election.cc:288] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 770d42b044b94d56a1ba645952e4496a (127.0.98.2:38951), 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293)
W20260620 09:19:07.640709 15237 leader_election.cc:341] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 770d42b044b94d56a1ba645952e4496a (127.0.98.2:38951): Not found: Tablet not found: 053c070ae0554db6aac14862714cb972
W20260620 09:19:07.640774 15237 leader_election.cc:341] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fa09c96387c49359a077279b4a33a5c (127.0.98.3:41293): Not found: Tablet not found: 053c070ae0554db6aac14862714cb972
I20260620 09:19:07.640787 15237 leader_election.cc:302] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c61fc3e9229f41b0b30a1e2e5adebafe; no voters: 6fa09c96387c49359a077279b4a33a5c, 770d42b044b94d56a1ba645952e4496a
I20260620 09:19:07.640851 15653 raft_consensus.cc:2726] T 053c070ae0554db6aac14862714cb972 P c61fc3e9229f41b0b30a1e2e5adebafe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:19:08.229346 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:08.229134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:52827)
0620 09:19:08.229262 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:08.229300 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260620 09:19:08.295197 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:08.294966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:58677)
0620 09:19:08.295110 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:08.295152 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
I20260620 09:19:08.414847 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:08.414596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:35319)
0620 09:19:08.414754 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:08.414803 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260620 09:19:09.230031 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:09.229773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:52007)
0620 09:19:09.229925 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:09.229977 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260620 09:19:09.295961 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:09.295723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:59585)
0620 09:19:09.295869 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:09.295915 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260620 09:19:09.415563 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:09.415296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:35467)
0620 09:19:09.415472 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:09.415517 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
I20260620 09:19:10.230731 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:10.230507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:53921)
0620 09:19:10.230643 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:10.230683 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260620 09:19:10.296681 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:10.296463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:36835)
0620 09:19:10.296596 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:10.296637 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260620 09:19:10.416205 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:10.415980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:35709)
0620 09:19:10.416127 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:10.416160 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260620 09:19:11.231478 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:11.231231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:44151)
0620 09:19:11.231396 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:11.231430 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260620 09:19:11.297425 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:11.297158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:45405)
0620 09:19:11.297336 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:11.297378 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":102,"threads_started":1}
I20260620 09:19:11.416877 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:11.416655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:45119)
0620 09:19:11.416800 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:11.416833 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260620 09:19:12.232108 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:12.231855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:55643)
0620 09:19:12.231982 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:12.232024 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260620 09:19:12.298127 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:12.297892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:34907)
0620 09:19:12.298037 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:12.298080 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260620 09:19:12.417563 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:12.417312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:37945)
0620 09:19:12.417436 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:12.417485 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260620 09:19:13.232918 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:13.232688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.1:43951)
0620 09:19:13.232828 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:13.232870 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260620 09:19:13.298979 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:13.298723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.2:36817)
0620 09:19:13.298863 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:13.298918 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260620 09:19:13.418195 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:13.417969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35695 (local address 127.0.98.3:60907)
0620 09:19:13.418109 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:13.418149 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260620 09:19:13.424552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 15191
I20260620 09:19:13.428849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 15355
I20260620 09:19:13.432613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 15487
2026-06-20T09:19:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18083 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:19:13 dist-test-slave-rq8c krb5kdc[15674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:19:13 dist-test-slave-rq8c krb5kdc[15674](info): set up 2 sockets
Jun 20 09:19:13 dist-test-slave-rq8c krb5kdc[15674](info): commencing operation
krb5kdc: starting...
W20260620 09:19:14.379437 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:19:14 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:19:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:19:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:19:14.487746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34225
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:19:14.566143 15687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:19:14.566341 15687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:19:14.566362 15687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:19:14.567466 15687 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 09:19:14.567498 15687 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 09:19:14.567509 15687 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 09:19:14.567519 15687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:19:14.567530 15687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:19:14.567543 15687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:19:14.567552 15687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:19:14.567569 15687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:19:14.569047 15687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:19:14.569237 15687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:19:14.569409 15687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:19:14.574251 15695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:14.574302 15693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:14.574301 15687 server_base.cc:1034] running on GCE node
W20260620 09:19:14.574272 15692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:14.574688 15687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:19:14.574990 15687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:19:14.576122 15687 hybrid_clock.cc:648] HybridClock initialized: now 1781947154576108 us; error 30 us; skew 500 ppm
Jun 20 09:19:14 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:19:14.578503 15687 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:19:14.578936 15687 webserver.cc:460] Webserver started at http://127.0.98.62:35923/ using document root <none> and password file <none>
I20260620 09:19:14.579141 15687 fs_manager.cc:359] Metadata directory not provided
I20260620 09:19:14.579190 15687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:19:14.579309 15687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:19:14.580368 15687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "fb098963851b4bcf96304a38400a198e"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "4826fcc9aabcedb4891a0e76f2b6a828"
server_key_iv: "b8a01f65908b9d315561e5f7c2e545ad"
server_key_version: "encryptionkey@0"
I20260620 09:19:14.580703 15687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb098963851b4bcf96304a38400a198e"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "4826fcc9aabcedb4891a0e76f2b6a828"
server_key_iv: "b8a01f65908b9d315561e5f7c2e545ad"
server_key_version: "encryptionkey@0"
I20260620 09:19:14.581956 15687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:19:14.582715 15702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:14.582932 15687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:19:14.583060 15687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "fb098963851b4bcf96304a38400a198e"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "4826fcc9aabcedb4891a0e76f2b6a828"
server_key_iv: "b8a01f65908b9d315561e5f7c2e545ad"
server_key_version: "encryptionkey@0"
I20260620 09:19:14.583127 15687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:19:14.592664 15687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:19:14.594955 15687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:19:14.595114 15687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:19:14.598788 15687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34225
I20260620 09:19:14.598830 15754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34225 every 8 connection(s)
I20260620 09:19:14.599143 15687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:19:14.599984 15755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:19:14.601871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 15687
I20260620 09:19:14.601958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:19:14.602136 392 external_mini_cluster.cc:1428] Setting key 620cd6e38096c79ea330245cd89c8202
I20260620 09:19:14.602339 15755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Bootstrap starting.
I20260620 09:19:14.602977 15755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Neither blocks nor log segments found. Creating new log.
I20260620 09:19:14.603256 15755 log.cc:824] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:19:14.603883 15755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: No bootstrap required, opened a new log
I20260620 09:19:14.605619 15755 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260620 09:19:14.605754 15755 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:19:14.605785 15755 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb098963851b4bcf96304a38400a198e, State: Initialized, Role: FOLLOWER
I20260620 09:19:14.605906 15755 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260620 09:19:14.605978 15755 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:19:14.606024 15755 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:19:14.606071 15755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:19:14.606578 15755 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260620 09:19:14.606699 15755 leader_election.cc:302] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb098963851b4bcf96304a38400a198e; no voters:
I20260620 09:19:14.606877 15755 leader_election.cc:288] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:19:14.606925 15760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:19:14.607082 15760 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 1 LEADER]: Becoming Leader. State: Replica: fb098963851b4bcf96304a38400a198e, State: Running, Role: LEADER
I20260620 09:19:14.607162 15755 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:19:14.607227 15760 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260620 09:19:14.607605 15760 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb098963851b4bcf96304a38400a198e. Latest consensus state: current_term: 1 leader_uuid: "fb098963851b4bcf96304a38400a198e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } } }
I20260620 09:19:14.607858 15760 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: This master's current role is: LEADER
I20260620 09:19:14.607800 15762 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb098963851b4bcf96304a38400a198e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } } }
I20260620 09:19:14.608227 15762 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: This master's current role is: LEADER
I20260620 09:19:14.608417 15769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 09:19:14 dist-test-slave-rq8c krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:19:14.609036 15769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:19:14.610960 15769 catalog_manager.cc:1347] Generated new cluster ID: 19b5a65d18c845e8846c4ed7fbc3ef01
I20260620 09:19:14.611043 15769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:19:14.612645 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:14.602607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.0.98.62:34225)
0620 09:19:14.602789 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:14.602797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:14.602821 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:19:14.603273 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:14.603280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:14.603326 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:14.603463 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:14.604093 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:14.604981 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:14.605736 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:14.606271 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:14.606364 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:14.609275 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:14.609279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:14.609279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:14.609291 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:14.611186 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:14.611504 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:14.611507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:14.611507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:14.611603 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:14.611808 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:14.611809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:14.611809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:14.612059 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:14.612195 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:14.612305 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 09:19:14.612436 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260620 09:19:14.653178 15769 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:19:14.653927 15769 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:19:14.659879 15769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Generated new TSK 0
I20260620 09:19:14.660079 15769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:19:14.684085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34225
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:19:14.763049 15781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:19:14.763265 15781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:19:14.763289 15781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:19:14.764467 15781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:19:14.764508 15781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:19:14.764534 15781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:19:14.765987 15781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:19:14.766212 15781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:19:14.766425 15781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:19:14.771221 15787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:14.771227 15789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:14.771227 15786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:14.771569 15781 server_base.cc:1034] running on GCE node
I20260620 09:19:14.771762 15781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:19:14.771971 15781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:19:14.773102 15781 hybrid_clock.cc:648] HybridClock initialized: now 1781947154773088 us; error 30 us; skew 500 ppm
Jun 20 09:19:14 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:19:14.775427 15781 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:19:14.775892 15781 webserver.cc:460] Webserver started at http://127.0.98.1:35837/ using document root <none> and password file <none>
I20260620 09:19:14.776104 15781 fs_manager.cc:359] Metadata directory not provided
I20260620 09:19:14.776153 15781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:19:14.776273 15781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:19:14.777275 15781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "23d1945502e14cc3939489b78fd55141"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "a8b2778a57d9bfa8292ae79016baccac"
server_key_iv: "a5f43373722b07195147e2a16b5e0e45"
server_key_version: "encryptionkey@0"
I20260620 09:19:14.777603 15781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "23d1945502e14cc3939489b78fd55141"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "a8b2778a57d9bfa8292ae79016baccac"
server_key_iv: "a5f43373722b07195147e2a16b5e0e45"
server_key_version: "encryptionkey@0"
I20260620 09:19:14.778779 15781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:19:14.779470 15796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:14.779646 15781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:14.779740 15781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "23d1945502e14cc3939489b78fd55141"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "a8b2778a57d9bfa8292ae79016baccac"
server_key_iv: "a5f43373722b07195147e2a16b5e0e45"
server_key_version: "encryptionkey@0"
I20260620 09:19:14.779798 15781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:19:14.787918 15781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:19:14.789974 15781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:19:14.790107 15781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:19:14.790311 15781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:19:14.790599 15781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:19:14.790632 15781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:14.790660 15781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:19:14.790683 15781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:14.796329 15781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41443
I20260620 09:19:14.796525 15909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41443 every 8 connection(s)
I20260620 09:19:14.796782 15781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:19:14.798213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 15781
I20260620 09:19:14.798321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:19:14.798539 392 external_mini_cluster.cc:1428] Setting key 82985da07df395820300cdba3c90e686
Jun 20 09:19:14 dist-test-slave-rq8c krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:19:14.806286 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:14.797539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34387 (local address 127.0.98.62:34225)
0620 09:19:14.797614 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:14.797616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:14.797757 (+ 141us) server_negotiation.cc:407] Connection header received
0620 09:19:14.798297 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:14.798298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:14.798309 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:14.798351 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:14.799494 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:14.800048 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:14.801145 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:14.801422 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:14.801491 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:14.803660 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:14.803661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:14.803661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:14.803667 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:14.804911 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:14.805353 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:14.805354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:14.805355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:14.805406 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:14.805669 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:14.805669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:14.805669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:14.805766 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:14.805811 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:14.806130 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 09:19:14.806207 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:19:14.806478 15912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:14.797509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:34387)
0620 09:19:14.797645 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:14.797679 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:14.798187 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:14.798444 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:14.798449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:14.799125 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:19:14.799410 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:14.799417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:14.800132 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:14.800133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:14.801057 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:14.801059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:14.801493 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:14.801493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:14.801589 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:14.802075 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:19:14.802090 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:19:14.803528 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:19:14.804990 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:14.804998 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:14.805001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:14.805285 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:14.805467 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:14.805468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:14.805468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:14.805611 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:14.805817 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:19:14.805823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:19:14.805984 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 09:19:14.806103 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 09:19:14.806267 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20260620 09:19:14.806991 15910 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34225
I20260620 09:19:14.807087 15910 heartbeater.cc:464] Registering TS with master...
I20260620 09:19:14.807255 15910 heartbeater.cc:511] Master 127.0.98.62:34225 requested a full tablet report, sending...
I20260620 09:19:14.807768 15719 ts_manager.cc:194] Registered new tserver with Master: 23d1945502e14cc3939489b78fd55141 (127.0.98.1:41443)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:19:14.808552 15719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:19:14.821488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34225
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:19:14.900765 15915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:19:14.900969 15915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:19:14.901000 15915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:19:14.902149 15915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:19:14.902194 15915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:19:14.902228 15915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:19:14.903689 15915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:19:14.903923 15915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:19:14.904131 15915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:19:14.908907 15923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:14.908901 15921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:14.908931 15920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:14.909083 15915 server_base.cc:1034] running on GCE node
I20260620 09:19:14.909385 15915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:19:14.909621 15915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:19:14.910777 15915 hybrid_clock.cc:648] HybridClock initialized: now 1781947154910763 us; error 30 us; skew 500 ppm
Jun 20 09:19:14 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:19:14.913201 15915 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:19:14.913623 15915 webserver.cc:460] Webserver started at http://127.0.98.2:37533/ using document root <none> and password file <none>
I20260620 09:19:14.913830 15915 fs_manager.cc:359] Metadata directory not provided
I20260620 09:19:14.913882 15915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:19:14.914003 15915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:19:14.915019 15915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d0f4f7d904d45e791d014789d3b632a"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "2ecb7289c8f999c8ed3ffce5e37111e9"
server_key_iv: "62fe539b1f7064dd70fcd39c7c03b5be"
server_key_version: "encryptionkey@0"
I20260620 09:19:14.915342 15915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d0f4f7d904d45e791d014789d3b632a"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "2ecb7289c8f999c8ed3ffce5e37111e9"
server_key_iv: "62fe539b1f7064dd70fcd39c7c03b5be"
server_key_version: "encryptionkey@0"
I20260620 09:19:14.916574 15915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:19:14.917194 15930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:14.917353 15915 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:19:14.917434 15915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "3d0f4f7d904d45e791d014789d3b632a"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "2ecb7289c8f999c8ed3ffce5e37111e9"
server_key_iv: "62fe539b1f7064dd70fcd39c7c03b5be"
server_key_version: "encryptionkey@0"
I20260620 09:19:14.917488 15915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:19:14.931262 15915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:19:14.934135 15915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:19:14.934340 15915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:19:14.934628 15915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:19:14.934984 15915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:19:14.935024 15915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:14.935051 15915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:19:14.935062 15915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:14.941763 15915 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36213
I20260620 09:19:14.941824 16043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36213 every 8 connection(s)
I20260620 09:19:14.942165 15915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:19:14.945804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 15915
I20260620 09:19:14.945883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:19:14.946074 392 external_mini_cluster.cc:1428] Setting key 04e158a3e2d3b3e2c715d6cfc95b3bc3
Jun 20 09:19:14 dist-test-slave-rq8c krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:19:14.951793 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:14.942943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34731 (local address 127.0.98.62:34225)
0620 09:19:14.943005 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:14.943007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:14.943210 (+ 203us) server_negotiation.cc:407] Connection header received
0620 09:19:14.943767 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:14.943768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:14.943778 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:14.943844 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:14.944794 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:14.945285 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:14.946369 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:14.946585 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:14.946656 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:14.949257 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:14.949259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:14.949259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:14.949267 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:14.950529 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:14.950889 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:14.950890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:14.950891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:14.950942 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:14.951164 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:14.951165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:14.951165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:14.951280 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:14.951334 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:14.951593 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 09:19:14.951703 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:19:14.951975 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:14.942921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:34731)
0620 09:19:14.943082 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:14.943126 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:14.943681 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:14.943906 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:14.943909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:14.944519 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:19:14.944725 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:14.944730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:14.945381 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:14.945382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:14.946282 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:14.946285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:14.946680 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:14.946681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:14.946807 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:14.947389 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:19:14.947400 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:19:14.949161 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:19:14.950605 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:14.950612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:14.950615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:14.950824 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:14.950993 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:14.950994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:14.950994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:14.951108 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:14.951332 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:19:14.951338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:19:14.951478 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 09:19:14.951565 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 09:19:14.951757 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260620 09:19:14.952494 16044 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34225
I20260620 09:19:14.952610 16044 heartbeater.cc:464] Registering TS with master...
I20260620 09:19:14.952811 16044 heartbeater.cc:511] Master 127.0.98.62:34225 requested a full tablet report, sending...
I20260620 09:19:14.953171 15719 ts_manager.cc:194] Registered new tserver with Master: 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213)
I20260620 09:19:14.953531 15719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34731
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:19:14.967896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34225
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:19:15.045989 16049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:19:15.046203 16049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:19:15.046234 16049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:19:15.047586 16049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:19:15.047683 16049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:19:15.047727 16049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:19:15.049259 16049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:19:15.049598 16049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:19:15.049844 16049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:19:15.054728 16054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:15.054728 16057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:15.054823 16055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:15.054911 16049 server_base.cc:1034] running on GCE node
I20260620 09:19:15.055150 16049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:19:15.055362 16049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:19:15.056506 16049 hybrid_clock.cc:648] HybridClock initialized: now 1781947155056492 us; error 31 us; skew 500 ppm
Jun 20 09:19:15 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:19:15.058862 16049 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:19:15.059296 16049 webserver.cc:460] Webserver started at http://127.0.98.3:43101/ using document root <none> and password file <none>
I20260620 09:19:15.059486 16049 fs_manager.cc:359] Metadata directory not provided
I20260620 09:19:15.059530 16049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:19:15.059660 16049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:19:15.060671 16049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "40a763392ee34bda87d2e78be56c9d3f"
format_stamp: "Formatted at 2026-06-20 09:19:15 on dist-test-slave-rq8c"
server_key: "388bed1bdda3bcf17969e992f866a8da"
server_key_iv: "42e9b5b6e9e7f3323786227c97f7f0a7"
server_key_version: "encryptionkey@0"
I20260620 09:19:15.060983 16049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40a763392ee34bda87d2e78be56c9d3f"
format_stamp: "Formatted at 2026-06-20 09:19:15 on dist-test-slave-rq8c"
server_key: "388bed1bdda3bcf17969e992f866a8da"
server_key_iv: "42e9b5b6e9e7f3323786227c97f7f0a7"
server_key_version: "encryptionkey@0"
I20260620 09:19:15.062179 16049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:19:15.062866 16064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:15.063004 16049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:19:15.063083 16049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "40a763392ee34bda87d2e78be56c9d3f"
format_stamp: "Formatted at 2026-06-20 09:19:15 on dist-test-slave-rq8c"
server_key: "388bed1bdda3bcf17969e992f866a8da"
server_key_iv: "42e9b5b6e9e7f3323786227c97f7f0a7"
server_key_version: "encryptionkey@0"
I20260620 09:19:15.063148 16049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:19:15.081882 16049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:19:15.083740 16049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:19:15.083870 16049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:19:15.084093 16049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:19:15.084398 16049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:19:15.084431 16049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:15.084455 16049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:19:15.084478 16049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:15.090018 16049 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39571
I20260620 09:19:15.090065 16177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39571 every 8 connection(s)
I20260620 09:19:15.090361 16049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:19:15.092068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 16049
I20260620 09:19:15.092167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:19:15.092358 392 external_mini_cluster.cc:1428] Setting key 12a1c731f78996db5343c3b8d24c82f0
Jun 20 09:19:15 dist-test-slave-rq8c krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:19:15.101325 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.090982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36493 (local address 127.0.98.62:34225)
0620 09:19:15.091035 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:15.091037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:15.091246 (+ 209us) server_negotiation.cc:407] Connection header received
0620 09:19:15.091976 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:15.091978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:15.091991 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:15.092133 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:15.093352 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.094099 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.095441 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.095768 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.095833 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.098790 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:15.098791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:15.098792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:15.098800 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:15.100289 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:15.100639 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:15.100640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:15.100640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:15.100685 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:15.100861 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:15.100861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:15.100861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:15.100958 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:15.101004 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:15.101203 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 09:19:15.101260 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:19:15.101522 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.090932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:36493)
0620 09:19:15.091118 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:15.091168 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:15.091842 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:15.092217 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:15.092222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:15.092970 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:19:15.093261 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:15.093269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.094191 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.094192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:15.095363 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:15.095366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.095864 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.095865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:15.096034 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.097022 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:19:15.097033 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:19:15.098708 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:19:15.100363 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:15.100368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:15.100370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:15.100587 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:15.100721 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:15.100721 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:15.100721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:15.100802 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:15.101001 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:19:15.101004 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:19:15.101117 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 09:19:15.101192 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:19:15.101305 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260620 09:19:15.101928 16178 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34225
I20260620 09:19:15.102012 16178 heartbeater.cc:464] Registering TS with master...
I20260620 09:19:15.102154 16178 heartbeater.cc:511] Master 127.0.98.62:34225 requested a full tablet report, sending...
I20260620 09:19:15.102463 15719 ts_manager.cc:194] Registered new tserver with Master: 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571)
I20260620 09:19:15.102825 15719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36493
I20260620 09:19:15.103801 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:19:15.110314 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.105060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.0.98.62:34225)
0620 09:19:15.105093 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:15.105095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:15.105176 (+ 81us) server_negotiation.cc:407] Connection header received
0620 09:19:15.105223 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:15.105224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:15.105231 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:15.105276 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:15.105753 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.106211 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.106933 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.107120 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.107166 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.108209 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:15.108210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:15.108210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:15.108214 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:15.109461 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:15.109755 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:15.109756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:15.109756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:15.109800 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:15.109922 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:15.109922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:15.109922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:15.110048 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:15.110115 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:15.110190 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 09:19:15.110247 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 09:19:15.111690 15719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:19:15.113149 15719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:19:15.118762 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.114989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39571 (local address 127.0.0.1:38796)
0620 09:19:15.115431 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:15.115439 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:15.115484 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:15.115665 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:15.115666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:15.115668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:19:15.115713 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:15.115715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.116590 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.116591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:15.117823 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:15.117826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.118480 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.118481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:15.118553 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.118576 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:19:15.118636 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:19:15.118687 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":56,"threads_started":1}
I20260620 09:19:15.118773 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.115146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.0.98.3:39571)
0620 09:19:15.115316 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:15.115321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:15.115494 (+ 173us) server_negotiation.cc:407] Connection header received
0620 09:19:15.115535 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:15.115537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:15.115551 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:15.115638 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:19:15.115791 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.116503 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.117911 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.118410 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.118470 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.118528 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:15.118655 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 09:19:15.118696 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260620 09:19:15.119019 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.114986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41443 (local address 127.0.0.1:35714)
0620 09:19:15.115139 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:15.115174 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:15.115224 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:15.116032 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:15.116032 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:15.116034 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:19:15.116077 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:15.116078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.117213 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.117213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:15.118046 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:15.118049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.118840 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.118841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:15.118891 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.118893 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:19:15.118918 (+ 25us) client_negotiation.cc:240] Negotiation successful
0620 09:19:15.118945 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":52,"threads_started":1}
I20260620 09:19:15.119002 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.114986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36213 (local address 127.0.0.1:44300)
0620 09:19:15.115311 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:15.115321 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:15.115370 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:15.115567 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:15.115570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:15.115572 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:19:15.115659 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:15.115661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.116904 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.116905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:15.117928 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:15.117930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.118788 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.118789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:15.118850 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.118854 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:19:15.118881 (+ 27us) client_negotiation.cc:240] Negotiation successful
0620 09:19:15.118919 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260620 09:19:15.119263 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.115432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.0.98.1:41443)
0620 09:19:15.115649 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:15.115653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:15.115663 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:19:15.115701 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:15.115703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:15.115718 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:15.116147 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:19:15.116216 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.117130 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.118126 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.118766 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.119057 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.119123 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:15.119151 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 09:19:15.119187 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260620 09:19:15.119530 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.115238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.0.98.2:36213)
0620 09:19:15.115393 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:15.115396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:15.115403 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:19:15.115435 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:15.115437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:15.115449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:15.115550 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:19:15.115743 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.116805 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.118000 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.118690 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.119282 (+ 592us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.119346 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:15.119369 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 09:19:15.119446 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260620 09:19:15.119781 16112 tablet_service.cc:1467] Processing CreateTablet for tablet bfb75317359a41ecaf03c93f3c0cbe7c (DEFAULT_TABLE table=test-table [id=dac856f3d0d24cfb8b8cb4e59610b855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:19:15.119935 15844 tablet_service.cc:1467] Processing CreateTablet for tablet bfb75317359a41ecaf03c93f3c0cbe7c (DEFAULT_TABLE table=test-table [id=dac856f3d0d24cfb8b8cb4e59610b855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:19:15.120127 16112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb75317359a41ecaf03c93f3c0cbe7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:19:15.120227 15844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb75317359a41ecaf03c93f3c0cbe7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:19:15.120117 15978 tablet_service.cc:1467] Processing CreateTablet for tablet bfb75317359a41ecaf03c93f3c0cbe7c (DEFAULT_TABLE table=test-table [id=dac856f3d0d24cfb8b8cb4e59610b855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:19:15.120393 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb75317359a41ecaf03c93f3c0cbe7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:19:15.123179 16194 tablet_bootstrap.cc:492] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f: Bootstrap starting.
I20260620 09:19:15.123378 16192 tablet_bootstrap.cc:492] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a: Bootstrap starting.
I20260620 09:19:15.123407 16193 tablet_bootstrap.cc:492] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Bootstrap starting.
I20260620 09:19:15.123883 16194 tablet_bootstrap.cc:654] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f: Neither blocks nor log segments found. Creating new log.
I20260620 09:19:15.123939 16192 tablet_bootstrap.cc:654] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a: Neither blocks nor log segments found. Creating new log.
I20260620 09:19:15.124069 16193 tablet_bootstrap.cc:654] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Neither blocks nor log segments found. Creating new log.
I20260620 09:19:15.124439 16192 log.cc:824] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:19:15.124454 16194 log.cc:824] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f: Log is configured to *not* fsync() on all Append() calls
I20260620 09:19:15.124595 16193 log.cc:824] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Log is configured to *not* fsync() on all Append() calls
I20260620 09:19:15.125080 16192 tablet_bootstrap.cc:492] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a: No bootstrap required, opened a new log
I20260620 09:19:15.125139 16192 ts_tablet_manager.cc:1397] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:19:15.125267 16193 tablet_bootstrap.cc:492] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: No bootstrap required, opened a new log
I20260620 09:19:15.125317 16193 ts_tablet_manager.cc:1397] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:19:15.126605 16192 raft_consensus.cc:348] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:15.126742 16192 raft_consensus.cc:374] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:19:15.126772 16192 raft_consensus.cc:729] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0f4f7d904d45e791d014789d3b632a, State: Initialized, Role: FOLLOWER
I20260620 09:19:15.126782 16193 raft_consensus.cc:348] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:15.126884 16192 consensus_queue.cc:260] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:15.126942 16193 raft_consensus.cc:374] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:19:15.126977 16193 raft_consensus.cc:729] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1945502e14cc3939489b78fd55141, State: Initialized, Role: FOLLOWER
I20260620 09:19:15.127091 16193 consensus_queue.cc:260] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:15.127135 16192 ts_tablet_manager.cc:1428] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 09:19:15.127239 16044 heartbeater.cc:503] Master 127.0.98.62:34225 was elected leader, sending a full tablet report...
I20260620 09:19:15.127331 16193 ts_tablet_manager.cc:1428] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:19:15.127342 15910 heartbeater.cc:503] Master 127.0.98.62:34225 was elected leader, sending a full tablet report...
I20260620 09:19:15.127646 16194 tablet_bootstrap.cc:492] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f: No bootstrap required, opened a new log
I20260620 09:19:15.127717 16194 ts_tablet_manager.cc:1397] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 09:19:15.129186 16194 raft_consensus.cc:348] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:15.129321 16194 raft_consensus.cc:374] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:19:15.129354 16194 raft_consensus.cc:729] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a763392ee34bda87d2e78be56c9d3f, State: Initialized, Role: FOLLOWER
I20260620 09:19:15.129460 16194 consensus_queue.cc:260] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:15.129840 16194 ts_tablet_manager.cc:1428] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:19:15.129894 16178 heartbeater.cc:503] Master 127.0.98.62:34225 was elected leader, sending a full tablet report...
I20260620 09:19:15.161405 16199 raft_consensus.cc:482] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:19:15.161520 16199 raft_consensus.cc:504] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:15.161827 16199 leader_election.cc:288] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213), 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571)
I20260620 09:19:15.164610 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.162030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59043 (local address 127.0.98.3:39571)
0620 09:19:15.162081 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:15.162083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:15.162096 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:19:15.162170 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:15.162170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:15.162181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:15.162223 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:19:15.162365 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.162907 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.163904 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.164298 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.164349 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.164358 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:15.164508 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 09:19:15.164544 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:19:15.164614 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.162033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46047 (local address 127.0.98.2:36213)
0620 09:19:15.162075 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:15.162077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:15.162331 (+ 254us) server_negotiation.cc:407] Connection header received
0620 09:19:15.162364 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:15.162365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:15.162377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:15.162436 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:19:15.162567 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.163030 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.163960 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.164323 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.164387 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.164396 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:15.164506 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 09:19:15.164543 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:19:15.164626 15912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.161977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39571 (local address 127.0.98.1:59043)
0620 09:19:15.162056 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:15.162068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:15.162107 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:15.162230 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:15.162231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:15.162232 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:19:15.162301 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:15.162303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.162977 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.162978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:15.163832 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:15.163835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.164366 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.164367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:15.164444 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.164447 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:19:15.164498 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 09:19:15.164545 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":4}
I20260620 09:19:15.164631 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.161977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36213 (local address 127.0.98.1:46047)
0620 09:19:15.162261 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:15.162270 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:15.162321 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:15.162442 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:15.162443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:15.162444 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:19:15.162510 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:15.162512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.163095 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.163096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:15.163879 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:15.163881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.164395 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.164395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:15.164449 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.164452 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:19:15.164491 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 09:19:15.164531 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":109,"thread_start_us":109,"threads_started":1}
I20260620 09:19:15.164937 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb75317359a41ecaf03c93f3c0cbe7c" candidate_uuid: "23d1945502e14cc3939489b78fd55141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0f4f7d904d45e791d014789d3b632a" is_pre_election: true
I20260620 09:19:15.164935 16132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb75317359a41ecaf03c93f3c0cbe7c" candidate_uuid: "23d1945502e14cc3939489b78fd55141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a763392ee34bda87d2e78be56c9d3f" is_pre_election: true
I20260620 09:19:15.165082 16132 raft_consensus.cc:2454] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d1945502e14cc3939489b78fd55141 in term 0.
I20260620 09:19:15.165082 15998 raft_consensus.cc:2454] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23d1945502e14cc3939489b78fd55141 in term 0.
I20260620 09:19:15.165282 15799 leader_election.cc:302] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1945502e14cc3939489b78fd55141, 3d0f4f7d904d45e791d014789d3b632a; no voters:
I20260620 09:19:15.165407 16199 raft_consensus.cc:2781] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:19:15.165467 16199 raft_consensus.cc:482] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:19:15.165483 16199 raft_consensus.cc:3037] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:19:15.166222 16199 raft_consensus.cc:504] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:15.166352 16199 leader_election.cc:288] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 1 election: Requested vote from peers 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213), 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571)
I20260620 09:19:15.166488 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb75317359a41ecaf03c93f3c0cbe7c" candidate_uuid: "23d1945502e14cc3939489b78fd55141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0f4f7d904d45e791d014789d3b632a"
I20260620 09:19:15.166494 16132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb75317359a41ecaf03c93f3c0cbe7c" candidate_uuid: "23d1945502e14cc3939489b78fd55141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a763392ee34bda87d2e78be56c9d3f"
I20260620 09:19:15.166559 15998 raft_consensus.cc:3037] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:19:15.166560 16132 raft_consensus.cc:3037] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:19:15.167234 16132 raft_consensus.cc:2454] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1945502e14cc3939489b78fd55141 in term 1.
I20260620 09:19:15.167277 15998 raft_consensus.cc:2454] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23d1945502e14cc3939489b78fd55141 in term 1.
I20260620 09:19:15.167408 15800 leader_election.cc:302] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1945502e14cc3939489b78fd55141, 40a763392ee34bda87d2e78be56c9d3f; no voters:
I20260620 09:19:15.167539 16199 raft_consensus.cc:2781] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:19:15.167690 16199 raft_consensus.cc:686] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 1 LEADER]: Becoming Leader. State: Replica: 23d1945502e14cc3939489b78fd55141, State: Running, Role: LEADER
I20260620 09:19:15.167785 16199 consensus_queue.cc:237] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:15.168547 15719 catalog_manager.cc:5495] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23d1945502e14cc3939489b78fd55141 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "23d1945502e14cc3939489b78fd55141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } health_report { overall_health: HEALTHY } } }
I20260620 09:19:15.175854 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.173896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.0.98.1:41443)
0620 09:19:15.173935 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:15.173937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:15.173957 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:19:15.173984 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:15.173985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:15.174001 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:15.174046 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:19:15.174140 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.174583 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.175247 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:15.175426 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:15.175469 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:15.175565 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:19:15.175699 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:19:15.175756 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:15.175773 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 09:19:15.175789 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:19:15.177400 15998 raft_consensus.cc:1261] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a [term 1 FOLLOWER]: Refusing update from remote peer 23d1945502e14cc3939489b78fd55141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:19:15.177403 16132 raft_consensus.cc:1261] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f [term 1 FOLLOWER]: Refusing update from remote peer 23d1945502e14cc3939489b78fd55141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:19:15.177727 16199 consensus_queue.cc:1035] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:19:15.177858 16202 consensus_queue.cc:1035] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:19:15.180022 16205 mvcc.cc:205] Tried to move back new op lower bound from 7298855547604418560 to 7298855547567927296. Current Snapshot: MvccSnapshot[applied={T|T < 7298855547604418560}]
I20260620 09:19:15.180496 16207 mvcc.cc:205] Tried to move back new op lower bound from 7298855547604418560 to 7298855547567927296. Current Snapshot: MvccSnapshot[applied={T|T < 7298855547604418560}]
I20260620 09:19:15.181011 16206 mvcc.cc:205] Tried to move back new op lower bound from 7298855547604418560 to 7298855547567927296. Current Snapshot: MvccSnapshot[applied={T|T < 7298855547604418560}]
I20260620 09:19:15.182535 15718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:19:15.182616 15718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:19:15.183313 15718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Sending DeleteTablet for 3 replicas of tablet bfb75317359a41ecaf03c93f3c0cbe7c
I20260620 09:19:15.183598 15844 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb75317359a41ecaf03c93f3c0cbe7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:19:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35714
I20260620 09:19:15.183670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 15781
I20260620 09:19:15.183666 15978 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb75317359a41ecaf03c93f3c0cbe7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:19:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44300
I20260620 09:19:15.184295 16112 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb75317359a41ecaf03c93f3c0cbe7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:19:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38796
I20260620 09:19:15.184465 16215 tablet_replica.cc:330] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f: stopping tablet replica
I20260620 09:19:15.184553 16215 raft_consensus.cc:2229] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:19:15.184626 16214 tablet_replica.cc:330] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a: stopping tablet replica
I20260620 09:19:15.184676 16215 raft_consensus.cc:2258] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:19:15.184731 16214 raft_consensus.cc:2229] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:19:15.184906 16214 raft_consensus.cc:2258] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:19:15.185114 16215 ts_tablet_manager.cc:1905] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:19:15.185313 16214 ts_tablet_manager.cc:1905] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:19:15.186661 16215 ts_tablet_manager.cc:1918] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:19:15.186731 16215 log.cc:1196] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/bfb75317359a41ecaf03c93f3c0cbe7c
I20260620 09:19:15.186808 16214 ts_tablet_manager.cc:1918] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:19:15.186892 16214 log.cc:1196] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/bfb75317359a41ecaf03c93f3c0cbe7c
I20260620 09:19:15.187021 16215 ts_tablet_manager.cc:1939] T bfb75317359a41ecaf03c93f3c0cbe7c P 40a763392ee34bda87d2e78be56c9d3f: Deleting consensus metadata
I20260620 09:19:15.187150 16214 ts_tablet_manager.cc:1939] T bfb75317359a41ecaf03c93f3c0cbe7c P 3d0f4f7d904d45e791d014789d3b632a: Deleting consensus metadata
I20260620 09:19:15.187430 15706 catalog_manager.cc:4841] TS 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571): tablet bfb75317359a41ecaf03c93f3c0cbe7c (table test-table [id=dac856f3d0d24cfb8b8cb4e59610b855]) successfully deleted
I20260620 09:19:15.187693 15705 catalog_manager.cc:4841] TS 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213): tablet bfb75317359a41ecaf03c93f3c0cbe7c (table test-table [id=dac856f3d0d24cfb8b8cb4e59610b855]) successfully deleted
W20260620 09:19:15.189347 15703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41443 (error 108)
I20260620 09:19:15.189531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 15915
I20260620 09:19:15.189787 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:15.189691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41443 (local address 127.0.0.1:35740)
0620 09:19:15.189732 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:15.189757 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 09:19:15.189894 15703 catalog_manager.cc:4584] TS 23d1945502e14cc3939489b78fd55141 (127.0.98.1:41443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfb75317359a41ecaf03c93f3c0cbe7c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41443: connect: Connection refused (error 111)
I20260620 09:19:15.194124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 16049
I20260620 09:19:15.197981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 15687
I20260620 09:19:35.202440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34225
--webserver_interface=127.0.98.62
--webserver_port=35923
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:19:35.290441 16217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:19:35.290684 16217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:19:35.290716 16217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:19:35.291970 16217 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 09:19:35.292006 16217 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 09:19:35.292023 16217 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 09:19:35.292037 16217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:19:35.292054 16217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:19:35.292070 16217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:19:35.292088 16217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:19:35.292111 16217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:19:35.293649 16217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:19:35.293921 16217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:19:35.294145 16217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:19:35.303661 16223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:35.303661 16224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:35.303704 16226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:35.304056 16217 server_base.cc:1034] running on GCE node
I20260620 09:19:35.304440 16217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:19:35.304797 16217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:19:35.305944 16217 hybrid_clock.cc:648] HybridClock initialized: now 1781947175305926 us; error 40 us; skew 500 ppm
Jun 20 09:19:35 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:19:35.308833 16217 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:19:35.309383 16217 webserver.cc:460] Webserver started at http://127.0.98.62:35923/ using document root <none> and password file <none>
I20260620 09:19:35.309623 16217 fs_manager.cc:359] Metadata directory not provided
I20260620 09:19:35.309679 16217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:19:35.311712 16217 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:19:35.312448 16233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:35.312638 16217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:19:35.312734 16217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "fb098963851b4bcf96304a38400a198e"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "4826fcc9aabcedb4891a0e76f2b6a828"
server_key_iv: "b8a01f65908b9d315561e5f7c2e545ad"
server_key_version: "encryptionkey@0"
I20260620 09:19:35.313227 16217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:19:35.336719 16217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:19:35.338795 16217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:19:35.338933 16217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:19:35.343545 16217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34225
I20260620 09:19:35.343650 16285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34225 every 8 connection(s)
I20260620 09:19:35.344028 16217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:19:35.345835 16286 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:19:35.346529 16286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Bootstrap starting.
I20260620 09:19:35.348330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 16217
I20260620 09:19:35.348726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41443
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34225
--webserver_port=35837
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:19:35.355818 16286 log.cc:824] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:19:35.357753 16286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:19:35.358000 16286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Bootstrap complete.
I20260620 09:19:35.360176 16286 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260620 09:19:35.360419 16286 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb098963851b4bcf96304a38400a198e, State: Initialized, Role: FOLLOWER
I20260620 09:19:35.360594 16286 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260620 09:19:35.360666 16286 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:19:35.360718 16286 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:19:35.360766 16286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:19:35.361752 16286 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260620 09:19:35.361850 16286 leader_election.cc:302] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb098963851b4bcf96304a38400a198e; no voters:
I20260620 09:19:35.362015 16286 leader_election.cc:288] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:19:35.362061 16290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:19:35.362176 16290 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 2 LEADER]: Becoming Leader. State: Replica: fb098963851b4bcf96304a38400a198e, State: Running, Role: LEADER
I20260620 09:19:35.362300 16286 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:19:35.362383 16290 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260620 09:19:35.362754 16292 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb098963851b4bcf96304a38400a198e. Latest consensus state: current_term: 2 leader_uuid: "fb098963851b4bcf96304a38400a198e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } } }
I20260620 09:19:35.362787 16291 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fb098963851b4bcf96304a38400a198e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } } }
I20260620 09:19:35.362926 16292 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: This master's current role is: LEADER
I20260620 09:19:35.362946 16291 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: This master's current role is: LEADER
I20260620 09:19:35.364336 16305 catalog_manager.cc:1259] Loaded cluster ID: 19b5a65d18c845e8846c4ed7fbc3ef01
I20260620 09:19:35.364380 16305 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: loading cluster ID for follower catalog manager: success
I20260620 09:19:35.365204 16305 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: acquiring CA information for follower catalog manager: success
I20260620 09:19:35.365556 16305 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:19:35.365645 16299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:19:35.366039 16299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:19:35.366122 16299 catalog_manager.cc:1259] Loaded cluster ID: 19b5a65d18c845e8846c4ed7fbc3ef01
I20260620 09:19:35.366158 16299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:19:35.366351 16299 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:19:35.366443 16299 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Loaded TSK: 0
I20260620 09:19:35.377436 16299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Generated new TSK 1
I20260620 09:19:35.377538 16299 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:19:35.439488 16288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:19:35.439746 16288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:19:35.439781 16288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:19:35.440974 16288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:19:35.441015 16288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:19:35.441057 16288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:19:35.442549 16288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35837
--tserver_master_addrs=127.0.98.62:34225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:19:35.442790 16288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:19:35.443002 16288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:19:35.448326 16316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:35.448349 16314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:35.448520 16313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:35.448945 16288 server_base.cc:1034] running on GCE node
I20260620 09:19:35.449150 16288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:19:35.449374 16288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:19:35.450601 16288 hybrid_clock.cc:648] HybridClock initialized: now 1781947175450583 us; error 43 us; skew 500 ppm
Jun 20 09:19:35 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:19:35.453701 16288 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:19:35.454404 16288 webserver.cc:460] Webserver started at http://127.0.98.1:35837/ using document root <none> and password file <none>
I20260620 09:19:35.454689 16288 fs_manager.cc:359] Metadata directory not provided
I20260620 09:19:35.454777 16288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:19:35.456689 16288 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:19:35.457527 16323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:35.457728 16288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:19:35.457839 16288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "23d1945502e14cc3939489b78fd55141"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "a8b2778a57d9bfa8292ae79016baccac"
server_key_iv: "a5f43373722b07195147e2a16b5e0e45"
server_key_version: "encryptionkey@0"
I20260620 09:19:35.458405 16288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:19:35.484157 16288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:19:35.486717 16288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:19:35.486886 16288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:19:35.487177 16288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:19:35.487777 16330 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:19:35.489452 16288 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:19:35.489521 16288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 09:19:35.489554 16288 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:19:35.490579 16288 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:19:35.490623 16288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 09:19:35.490720 16330 tablet_bootstrap.cc:492] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Bootstrap starting.
I20260620 09:19:35.498685 16288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41443
I20260620 09:19:35.498924 16437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41443 every 8 connection(s)
I20260620 09:19:35.499212 16288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:19:35.503208 16330 log.cc:824] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Log is configured to *not* fsync() on all Append() calls
I20260620 09:19:35.505120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 16288
I20260620 09:19:35.505424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36213
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34225
--webserver_port=37533
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:19:35.506734 16330 tablet_bootstrap.cc:492] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:19:35.507043 16330 tablet_bootstrap.cc:492] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Bootstrap complete.
I20260620 09:19:35.507949 16330 ts_tablet_manager.cc:1397] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.013s
I20260620 09:19:35.509057 16330 raft_consensus.cc:348] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:35.509325 16330 raft_consensus.cc:729] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1945502e14cc3939489b78fd55141, State: Initialized, Role: FOLLOWER
I20260620 09:19:35.509490 16330 consensus_queue.cc:260] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:35.509827 16330 ts_tablet_manager.cc:1428] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Jun 20 09:19:35 dist-test-slave-rq8c krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:19:35.514879 16441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.503934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:35459)
0620 09:19:35.504153 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:35.504212 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:35.504915 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:35.505603 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:35.505607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:35.505993 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:19:35.506295 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.506304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.507870 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.507872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.508910 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.508913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.509502 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.509503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.509646 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.510244 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:19:35.510253 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:19:35.511596 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:19:35.513385 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:35.513392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:35.513394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:35.513604 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:35.513818 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:35.513819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:35.513819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:35.513922 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:35.514255 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:19:35.514261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:19:35.514397 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 09:19:35.514485 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:19:35.514665 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260620 09:19:35.514878 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.504038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35459 (local address 127.0.98.62:34225)
0620 09:19:35.504272 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:35.504282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:35.504325 (+ 43us) server_negotiation.cc:407] Connection header received
0620 09:19:35.505246 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:35.505254 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:35.505314 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:35.505455 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:35.506427 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.507732 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.508988 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.509401 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.509959 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.511712 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:35.511717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:35.511717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:35.511729 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:35.513307 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:35.513674 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:35.513677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:35.513678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:35.513760 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:35.513975 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:35.513976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:35.513976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:35.514193 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:35.514263 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:35.514542 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 09:19:35.514666 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":113,"threads_started":1}
I20260620 09:19:35.515769 16438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34225
I20260620 09:19:35.515888 16438 heartbeater.cc:464] Registering TS with master...
I20260620 09:19:35.516145 16438 heartbeater.cc:511] Master 127.0.98.62:34225 requested a full tablet report, sending...
I20260620 09:19:35.517108 16250 ts_manager.cc:194] Registered new tserver with Master: 23d1945502e14cc3939489b78fd55141 (127.0.98.1:41443)
I20260620 09:19:35.518198 16250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35459
I20260620 09:19:35.519276 16438 heartbeater.cc:503] Master 127.0.98.62:34225 was elected leader, sending a full tablet report...
Jun 20 09:19:35 dist-test-slave-rq8c krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:19:35.526120 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.517868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.0.98.1:41443)
0620 09:19:35.518282 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:35.518287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:35.518297 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:19:35.518329 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:35.518331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:35.518347 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:35.518418 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:35.518907 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.519754 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.520509 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.520808 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.520867 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.522889 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:35.522897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:35.522897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:35.522905 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:35.524661 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:35.525186 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:35.525189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:35.525190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:35.525257 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:35.525503 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:35.525505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:35.525505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:35.525676 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:35.525761 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:35.525957 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 09:19:35.526042 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":125,"threads_started":1}
I20260620 09:19:35.526193 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.517785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41443 (local address 127.0.0.1:38330)
0620 09:19:35.518038 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:35.518070 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:35.518148 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:35.518447 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:35.518454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:35.518767 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:19:35.518818 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.518820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.519821 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.519822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.520432 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.520435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.520870 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.520874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.520951 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.521398 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:19:35.521403 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:19:35.522796 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:19:35.524759 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:35.524766 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:35.524768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:35.525073 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:35.525335 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:35.525335 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:35.525336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:35.525411 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:35.525754 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:19:35.525761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:19:35.525828 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 09:19:35.525917 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 09:19:35.526067 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":113,"threads_started":1}
I20260620 09:19:35.526674 16372 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb75317359a41ecaf03c93f3c0cbe7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:19:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38330
I20260620 09:19:35.527371 16447 tablet_replica.cc:330] stopping tablet replica
I20260620 09:19:35.527498 16447 raft_consensus.cc:2229] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:19:35.527586 16447 raft_consensus.cc:2258] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:19:35.527995 16447 ts_tablet_manager.cc:1905] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:19:35.530192 16447 ts_tablet_manager.cc:1918] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:19:35.530267 16447 log.cc:1196] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal/wals/bfb75317359a41ecaf03c93f3c0cbe7c
I20260620 09:19:35.530452 16447 ts_tablet_manager.cc:1939] T bfb75317359a41ecaf03c93f3c0cbe7c P 23d1945502e14cc3939489b78fd55141: Deleting consensus metadata
I20260620 09:19:35.530943 16234 catalog_manager.cc:4841] TS 23d1945502e14cc3939489b78fd55141 (127.0.98.1:41443): tablet bfb75317359a41ecaf03c93f3c0cbe7c (table test-table [id=dac856f3d0d24cfb8b8cb4e59610b855]) successfully deleted
W20260620 09:19:35.597856 16443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:19:35.598079 16443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:19:35.598101 16443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:19:35.599273 16443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:19:35.599309 16443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:19:35.599334 16443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:19:35.600840 16443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37533
--tserver_master_addrs=127.0.98.62:34225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:19:35.601039 16443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:19:35.601235 16443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:19:35.606704 16453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:35.606712 16455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:35.606866 16443 server_base.cc:1034] running on GCE node
W20260620 09:19:35.606992 16452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:35.607285 16443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:19:35.607509 16443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:19:35.608647 16443 hybrid_clock.cc:648] HybridClock initialized: now 1781947175608646 us; error 47 us; skew 500 ppm
Jun 20 09:19:35 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:19:35.611302 16443 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:19:35.611806 16443 webserver.cc:460] Webserver started at http://127.0.98.2:37533/ using document root <none> and password file <none>
I20260620 09:19:35.612037 16443 fs_manager.cc:359] Metadata directory not provided
I20260620 09:19:35.612107 16443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:19:35.613534 16443 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:19:35.614173 16462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:35.614377 16443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:19:35.614460 16443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "3d0f4f7d904d45e791d014789d3b632a"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "2ecb7289c8f999c8ed3ffce5e37111e9"
server_key_iv: "62fe539b1f7064dd70fcd39c7c03b5be"
server_key_version: "encryptionkey@0"
I20260620 09:19:35.614827 16443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:19:35.622413 16443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:19:35.624554 16443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:19:35.624675 16443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:19:35.624883 16443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:19:35.625202 16443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:19:35.625236 16443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:35.625263 16443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:19:35.625279 16443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:35.631996 16443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36213
I20260620 09:19:35.632071 16575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36213 every 8 connection(s)
I20260620 09:19:35.632429 16443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:19:35.632920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 16443
I20260620 09:19:35.633244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39571
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34225
--webserver_port=43101
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:19:35 dist-test-slave-rq8c krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:19:35.642124 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.633271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58897 (local address 127.0.98.62:34225)
0620 09:19:35.633328 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:35.633330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:35.633794 (+ 464us) server_negotiation.cc:407] Connection header received
0620 09:19:35.634373 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:35.634375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:35.634391 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:35.634450 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:35.635485 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.636045 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.637088 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.637291 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.637344 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.639552 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:35.639554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:35.639554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:35.639561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:35.640910 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:35.641267 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:35.641268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:35.641269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:35.641318 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:35.641528 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:35.641529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:35.641529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:35.641635 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:35.641689 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:35.641943 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 09:19:35.642029 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:19:35.642354 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.633286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:58897)
0620 09:19:35.633616 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:35.633672 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:35.634264 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:35.634530 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:35.634535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:35.635166 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:19:35.635392 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.635396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.636127 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.636128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.637016 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.637018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.637358 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.637365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.637496 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.638088 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:19:35.638097 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:19:35.639468 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:19:35.640979 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:35.640985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:35.640988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:35.641205 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:35.641375 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:35.641376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:35.641376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:35.641475 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:35.641686 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:19:35.641692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:19:35.641826 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 09:19:35.641917 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 09:19:35.642071 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":102,"threads_started":1}
I20260620 09:19:35.642817 16576 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34225
I20260620 09:19:35.642946 16576 heartbeater.cc:464] Registering TS with master...
I20260620 09:19:35.643185 16576 heartbeater.cc:511] Master 127.0.98.62:34225 requested a full tablet report, sending...
I20260620 09:19:35.643589 16250 ts_manager.cc:194] Registered new tserver with Master: 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213)
I20260620 09:19:35.643990 16250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58897
W20260620 09:19:35.720470 16579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:19:35.720695 16579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:19:35.720726 16579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:19:35.721930 16579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:19:35.721971 16579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:19:35.722007 16579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:19:35.723479 16579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43101
--tserver_master_addrs=127.0.98.62:34225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:19:35.723742 16579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:19:35.723963 16579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:19:35.729254 16587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:35.729285 16584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:35.729286 16585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:35.729619 16579 server_base.cc:1034] running on GCE node
I20260620 09:19:35.729822 16579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:19:35.730073 16579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:19:35.731226 16579 hybrid_clock.cc:648] HybridClock initialized: now 1781947175731208 us; error 39 us; skew 500 ppm
Jun 20 09:19:35 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:19:35.734061 16579 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:19:35.734549 16579 webserver.cc:460] Webserver started at http://127.0.98.3:43101/ using document root <none> and password file <none>
I20260620 09:19:35.734753 16579 fs_manager.cc:359] Metadata directory not provided
I20260620 09:19:35.734818 16579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:19:35.736316 16579 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:19:35.736933 16594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:35.737112 16579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:19:35.737201 16579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "40a763392ee34bda87d2e78be56c9d3f"
format_stamp: "Formatted at 2026-06-20 09:19:15 on dist-test-slave-rq8c"
server_key: "388bed1bdda3bcf17969e992f866a8da"
server_key_iv: "42e9b5b6e9e7f3323786227c97f7f0a7"
server_key_version: "encryptionkey@0"
I20260620 09:19:35.737538 16579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:19:35.747330 16579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:19:35.749359 16579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:19:35.749488 16579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:19:35.749691 16579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:19:35.749991 16579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:19:35.750025 16579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:35.750049 16579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:19:35.750077 16579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:35.756681 16579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39571
I20260620 09:19:35.756747 16707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39571 every 8 connection(s)
I20260620 09:19:35.757046 16579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:19:35.758670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 16579
Jun 20 09:19:35 dist-test-slave-rq8c krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:19:35.765525 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.759094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.0.98.62:34225)
0620 09:19:35.759204 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:35.759206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:35.759367 (+ 161us) server_negotiation.cc:407] Connection header received
0620 09:19:35.759446 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:35.759447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:35.759459 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:35.759516 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:35.760139 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.760690 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.761448 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.761649 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.761712 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.763012 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:35.763014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:35.763015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:35.763021 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:35.764532 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:35.764823 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:35.764824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:35.764825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:35.764877 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:35.765051 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:35.765051 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:35.765052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:35.765159 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:35.765315 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:35.765349 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 09:19:35.765425 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
I20260620 09:19:35.766968 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.757845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38327 (local address 127.0.98.62:34225)
0620 09:19:35.757920 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:35.757923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:35.758129 (+ 206us) server_negotiation.cc:407] Connection header received
0620 09:19:35.758719 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:35.758720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:35.758734 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:35.758788 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:35.759860 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.760407 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.761671 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.761952 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.762024 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.764606 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:35.764607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:35.764608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:35.764614 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:35.765950 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:35.766317 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:35.766318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:35.766318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:35.766367 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:35.766536 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:35.766536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:35.766537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:35.766617 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:35.766657 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:35.766862 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 09:19:35.766916 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:19:35.767211 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.757821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:38327)
0620 09:19:35.758006 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:35.758043 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:35.758577 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:35.758867 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:35.758872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:35.759541 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:19:35.759776 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.759781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.760492 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.760493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.761504 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.761512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.762044 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.762045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.762253 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.762760 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:19:35.762769 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:19:35.764520 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:19:35.766012 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:35.766018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:35.766020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:35.766271 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:35.766414 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:35.766414 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:35.766415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:35.766503 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:35.766650 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:19:35.766653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:19:35.766771 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 09:19:35.766847 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:19:35.766983 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
I20260620 09:19:35.767587 16708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34225
I20260620 09:19:35.767702 16708 heartbeater.cc:464] Registering TS with master...
I20260620 09:19:35.767881 16708 heartbeater.cc:511] Master 127.0.98.62:34225 requested a full tablet report, sending...
I20260620 09:19:35.768242 16250 ts_manager.cc:194] Registered new tserver with Master: 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571)
I20260620 09:19:35.768616 16250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38327
I20260620 09:19:35.777089 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:19:35.779901 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.777453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.0.98.62:34225)
0620 09:19:35.777506 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:35.777509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:35.777621 (+ 112us) server_negotiation.cc:407] Connection header received
0620 09:19:35.777696 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:35.777697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:35.777744 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:35.777806 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:19:35.777939 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.778442 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.779180 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.779379 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.779431 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.779573 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:19:35.779709 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:19:35.779776 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:35.779811 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 09:19:35.779830 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:19:35.780184 16250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:19:35.781752 16250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:19:35.783785 16372 tablet_service.cc:1467] Processing CreateTablet for tablet 86b9a160a45246fb9dbe9025f58559e0 (DEFAULT_TABLE table=test-table [id=1675cae7d4db442e947bf2723f21cc4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:19:35.783962 16372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b9a160a45246fb9dbe9025f58559e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:19:35.785247 16330 tablet_bootstrap.cc:492] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: Bootstrap starting.
I20260620 09:19:35.785589 16330 tablet_bootstrap.cc:654] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: Neither blocks nor log segments found. Creating new log.
I20260620 09:19:35.786113 16330 tablet_bootstrap.cc:492] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: No bootstrap required, opened a new log
I20260620 09:19:35.786162 16330 ts_tablet_manager.cc:1397] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:19:35.786269 16330 raft_consensus.cc:348] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:35.786329 16330 raft_consensus.cc:374] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:19:35.786369 16330 raft_consensus.cc:729] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1945502e14cc3939489b78fd55141, State: Initialized, Role: FOLLOWER
I20260620 09:19:35.786417 16330 consensus_queue.cc:260] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:35.786499 16330 ts_tablet_manager.cc:1428] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:35.787246 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.783443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36213 (local address 127.0.0.1:46542)
0620 09:19:35.783695 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:35.783703 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:35.783745 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:35.784037 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:35.784038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:35.784040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:19:35.784086 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.784088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.785030 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.785031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.786057 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.786060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.787067 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.787067 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.787115 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.787129 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 09:19:35.787167 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 09:19:35.787198 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
I20260620 09:19:35.787292 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.783704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.0.98.2:36213)
0620 09:19:35.783878 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:35.783883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:35.783892 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:19:35.783929 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:35.783931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:35.783947 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:35.784015 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:19:35.784135 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.784951 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.786138 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.787003 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.787070 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.787125 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:35.787179 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 09:19:35.787215 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260620 09:19:35.787945 16510 tablet_service.cc:1467] Processing CreateTablet for tablet 86b9a160a45246fb9dbe9025f58559e0 (DEFAULT_TABLE table=test-table [id=1675cae7d4db442e947bf2723f21cc4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:19:35.788254 16510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b9a160a45246fb9dbe9025f58559e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:19:35.788395 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.783398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39571 (local address 127.0.0.1:49934)
0620 09:19:35.783440 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:35.783452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:35.783489 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:35.785452 (+ 1963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:35.785453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:35.785455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:19:35.785498 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.785499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.786564 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.786565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.787518 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.787521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.788154 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.788155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.788222 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.788227 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:19:35.788289 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 09:19:35.788329 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 09:19:35.788424 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.783519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.0.98.3:39571)
0620 09:19:35.785286 (+ 1767us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:35.785290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:35.785301 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:19:35.785352 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:35.785354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:35.785376 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:35.785453 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:19:35.785541 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.786477 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.787594 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.788089 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.788153 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.788204 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:35.788298 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 09:19:35.788334 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1737,"thread_start_us":108,"threads_started":1}
I20260620 09:19:35.789043 16642 tablet_service.cc:1467] Processing CreateTablet for tablet 86b9a160a45246fb9dbe9025f58559e0 (DEFAULT_TABLE table=test-table [id=1675cae7d4db442e947bf2723f21cc4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:19:35.789383 16642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b9a160a45246fb9dbe9025f58559e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:19:35.791756 16718 tablet_bootstrap.cc:492] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a: Bootstrap starting.
I20260620 09:19:35.792383 16718 tablet_bootstrap.cc:654] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a: Neither blocks nor log segments found. Creating new log.
I20260620 09:19:35.792508 16719 tablet_bootstrap.cc:492] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f: Bootstrap starting.
I20260620 09:19:35.792912 16718 log.cc:824] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:19:35.793059 16719 tablet_bootstrap.cc:654] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f: Neither blocks nor log segments found. Creating new log.
I20260620 09:19:35.793494 16719 log.cc:824] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f: Log is configured to *not* fsync() on all Append() calls
I20260620 09:19:35.793536 16718 tablet_bootstrap.cc:492] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a: No bootstrap required, opened a new log
I20260620 09:19:35.793579 16718 ts_tablet_manager.cc:1397] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:19:35.794075 16719 tablet_bootstrap.cc:492] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f: No bootstrap required, opened a new log
I20260620 09:19:35.794126 16719 ts_tablet_manager.cc:1397] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:19:35.795056 16718 raft_consensus.cc:348] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:35.795197 16718 raft_consensus.cc:374] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:19:35.795225 16718 raft_consensus.cc:729] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0f4f7d904d45e791d014789d3b632a, State: Initialized, Role: FOLLOWER
I20260620 09:19:35.795333 16718 consensus_queue.cc:260] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:35.795588 16718 ts_tablet_manager.cc:1428] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 09:19:35.795603 16576 heartbeater.cc:503] Master 127.0.98.62:34225 was elected leader, sending a full tablet report...
I20260620 09:19:35.795639 16719 raft_consensus.cc:348] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:35.795791 16719 raft_consensus.cc:374] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:19:35.795830 16719 raft_consensus.cc:729] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a763392ee34bda87d2e78be56c9d3f, State: Initialized, Role: FOLLOWER
I20260620 09:19:35.795948 16719 consensus_queue.cc:260] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:35.796205 16719 ts_tablet_manager.cc:1428] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:19:35.796211 16708 heartbeater.cc:503] Master 127.0.98.62:34225 was elected leader, sending a full tablet report...
W20260620 09:19:35.883040 16577 tablet.cc:2369] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:19:35.962085 16722 raft_consensus.cc:482] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:19:35.962237 16722 raft_consensus.cc:504] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:35.962589 16722 leader_election.cc:288] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571), 23d1945502e14cc3939489b78fd55141 (127.0.98.1:41443)
I20260620 09:19:35.965890 16724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.962787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39571 (local address 127.0.98.2:50059)
0620 09:19:35.963011 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:35.963020 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:35.963080 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:35.963220 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:35.963221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:35.963223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:19:35.963304 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.963305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.964001 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.964002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.965091 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.965094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.965706 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.965707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.965757 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.965761 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:19:35.965799 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 09:19:35.965831 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":47,"thread_start_us":101,"threads_started":1}
I20260620 09:19:35.965919 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.962837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50059 (local address 127.0.98.3:39571)
0620 09:19:35.962950 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:35.962952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:35.963083 (+ 131us) server_negotiation.cc:407] Connection header received
0620 09:19:35.963145 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:35.963146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:35.963158 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:35.963211 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:19:35.963363 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.963937 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.965165 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.965632 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.965701 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.965711 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:35.965814 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 09:19:35.965850 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 09:19:35.966336 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.962775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41443 (local address 127.0.98.2:59337)
0620 09:19:35.962882 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:35.962895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:35.962956 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:35.963138 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:35.963139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:35.963141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:19:35.963234 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.963236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.964408 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.964409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.965343 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:35.965345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.966116 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.966117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:35.966196 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.966200 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:19:35.966251 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 09:19:35.966292 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 09:19:35.966320 16662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b9a160a45246fb9dbe9025f58559e0" candidate_uuid: "3d0f4f7d904d45e791d014789d3b632a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a763392ee34bda87d2e78be56c9d3f" is_pre_election: true
I20260620 09:19:35.966377 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:35.962885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59337 (local address 127.0.98.1:41443)
0620 09:19:35.962935 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:35.962937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:35.962948 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:19:35.963022 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:35.963024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:35.963052 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:35.963138 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:19:35.963284 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.964333 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.965415 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:35.965963 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:35.966023 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:35.966088 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:35.966276 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 09:19:35.966318 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:19:35.966465 16662 raft_consensus.cc:2454] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0f4f7d904d45e791d014789d3b632a in term 0.
I20260620 09:19:35.966604 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b9a160a45246fb9dbe9025f58559e0" candidate_uuid: "3d0f4f7d904d45e791d014789d3b632a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1945502e14cc3939489b78fd55141" is_pre_election: true
I20260620 09:19:35.966660 16466 leader_election.cc:302] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0f4f7d904d45e791d014789d3b632a, 40a763392ee34bda87d2e78be56c9d3f; no voters:
I20260620 09:19:35.966754 16392 raft_consensus.cc:2454] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0f4f7d904d45e791d014789d3b632a in term 0.
I20260620 09:19:35.966914 16722 raft_consensus.cc:2781] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:19:35.966972 16722 raft_consensus.cc:482] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:19:35.967003 16722 raft_consensus.cc:3037] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:19:35.967705 16722 raft_consensus.cc:504] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:35.967823 16722 leader_election.cc:288] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [CANDIDATE]: Term 1 election: Requested vote from peers 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571), 23d1945502e14cc3939489b78fd55141 (127.0.98.1:41443)
I20260620 09:19:35.967948 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b9a160a45246fb9dbe9025f58559e0" candidate_uuid: "3d0f4f7d904d45e791d014789d3b632a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d1945502e14cc3939489b78fd55141"
I20260620 09:19:35.967971 16662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b9a160a45246fb9dbe9025f58559e0" candidate_uuid: "3d0f4f7d904d45e791d014789d3b632a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a763392ee34bda87d2e78be56c9d3f"
I20260620 09:19:35.968027 16662 raft_consensus.cc:3037] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:19:35.968021 16392 raft_consensus.cc:3037] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:19:35.968640 16662 raft_consensus.cc:2454] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0f4f7d904d45e791d014789d3b632a in term 1.
I20260620 09:19:35.968640 16392 raft_consensus.cc:2454] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0f4f7d904d45e791d014789d3b632a in term 1.
I20260620 09:19:35.968786 16463 leader_election.cc:302] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d1945502e14cc3939489b78fd55141, 3d0f4f7d904d45e791d014789d3b632a; no voters:
I20260620 09:19:35.968881 16722 raft_consensus.cc:2781] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:19:35.969003 16722 raft_consensus.cc:686] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 1 LEADER]: Becoming Leader. State: Replica: 3d0f4f7d904d45e791d014789d3b632a, State: Running, Role: LEADER
I20260620 09:19:35.969077 16722 consensus_queue.cc:237] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:35.969848 16250 catalog_manager.cc:5495] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3d0f4f7d904d45e791d014789d3b632a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } health_report { overall_health: UNKNOWN } } }
W20260620 09:19:36.000022 16439 tablet.cc:2369] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:19:36.007597 16709 tablet.cc:2369] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:19:36.016474 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:36.013924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.0.98.2:36213)
0620 09:19:36.014002 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:36.014004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:36.014028 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:19:36.014093 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:36.014094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:36.014118 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:36.014171 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:19:36.014274 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:36.014751 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:36.015590 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:36.015830 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:36.015899 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:36.016033 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:19:36.016081 (+ 48us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 09:19:36.016358 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46548: token expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 09:19:36.016541 16716 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46548: token expired
I20260620 09:19:36.023274 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:36.016620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.0.98.62:34225)
0620 09:19:36.016661 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:36.016662 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:36.017012 (+ 350us) server_negotiation.cc:407] Connection header received
0620 09:19:36.017078 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:36.017079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:36.017091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:36.017151 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:36.017678 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:36.018422 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:36.019363 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:36.019592 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:36.019688 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:36.020805 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:36.020806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:36.020807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:36.020813 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:36.022307 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:36.022602 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:36.022602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:36.022603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:36.022656 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:36.022782 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:36.022783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:36.022783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:36.022896 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:36.022985 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:36.023140 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 09:19:36.023202 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:19:36.026077 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:36.024138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.0.98.2:36213)
0620 09:19:36.024190 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:36.024193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:36.024200 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:19:36.024221 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:36.024222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:36.024234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:36.024303 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:19:36.024390 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:36.024899 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:36.025528 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:36.025699 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:36.025743 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:36.025856 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:19:36.025940 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:19:36.025976 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:36.026008 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 09:19:36.026025 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:19:36.027783 16392 raft_consensus.cc:1261] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Refusing update from remote peer 3d0f4f7d904d45e791d014789d3b632a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:19:36.027812 16662 raft_consensus.cc:1261] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f [term 1 FOLLOWER]: Refusing update from remote peer 3d0f4f7d904d45e791d014789d3b632a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:19:36.028106 16725 consensus_queue.cc:1035] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:19:36.028232 16722 consensus_queue.cc:1035] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [LEADER]: Connected to new peer: Peer: permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:19:36.030367 16728 mvcc.cc:205] Tried to move back new op lower bound from 7298855633007296512 to 7298855632770035712. Current Snapshot: MvccSnapshot[applied={T|T < 7298855633007296512 or (T in {7298855633007296512})}]
I20260620 09:19:36.030411 16729 mvcc.cc:205] Tried to move back new op lower bound from 7298855633007296512 to 7298855632770035712. Current Snapshot: MvccSnapshot[applied={T|T < 7298855633007296512}]
I20260620 09:19:36.030535 16730 mvcc.cc:205] Tried to move back new op lower bound from 7298855633007296512 to 7298855632770035712. Current Snapshot: MvccSnapshot[applied={T|T < 7298855633007296512}]
I20260620 09:19:36.032367 16249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:19:36.032444 16249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:19:36.033125 16249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Sending DeleteTablet for 3 replicas of tablet 86b9a160a45246fb9dbe9025f58559e0
I20260620 09:19:36.033396 16642 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b9a160a45246fb9dbe9025f58559e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:19:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49934
I20260620 09:19:36.033420 16510 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b9a160a45246fb9dbe9025f58559e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:19:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46542
I20260620 09:19:36.033515 16372 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b9a160a45246fb9dbe9025f58559e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:19:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38330
I20260620 09:19:36.033649 16739 tablet_replica.cc:330] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: stopping tablet replica
I20260620 09:19:36.033710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 16288
I20260620 09:19:36.033718 16739 raft_consensus.cc:2229] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:19:36.033775 16739 raft_consensus.cc:2258] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:19:36.034644 16737 tablet_replica.cc:330] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f: stopping tablet replica
I20260620 09:19:36.034731 16737 raft_consensus.cc:2229] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:19:36.034818 16737 raft_consensus.cc:2258] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:19:36.033735 16738 tablet_replica.cc:330] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a: stopping tablet replica
I20260620 09:19:36.036201 16738 raft_consensus.cc:2229] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:19:36.036324 16738 raft_consensus.cc:2258] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:19:36.036684 16738 ts_tablet_manager.cc:1905] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:19:36.038338 16738 ts_tablet_manager.cc:1918] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:19:36.038408 16738 log.cc:1196] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/86b9a160a45246fb9dbe9025f58559e0
I20260620 09:19:36.038661 16738 ts_tablet_manager.cc:1939] T 86b9a160a45246fb9dbe9025f58559e0 P 3d0f4f7d904d45e791d014789d3b632a: Deleting consensus metadata
I20260620 09:19:36.039039 16236 catalog_manager.cc:4841] TS 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213): tablet 86b9a160a45246fb9dbe9025f58559e0 (table test-table [id=1675cae7d4db442e947bf2723f21cc4d]) successfully deleted
I20260620 09:19:36.039947 16737 ts_tablet_manager.cc:1905] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 09:19:36.040309 16234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41443 (error 108)
I20260620 09:19:36.040987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 16443
I20260620 09:19:36.041625 16737 ts_tablet_manager.cc:1918] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:19:36.041699 16737 log.cc:1196] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/86b9a160a45246fb9dbe9025f58559e0
I20260620 09:19:36.041951 16737 ts_tablet_manager.cc:1939] T 86b9a160a45246fb9dbe9025f58559e0 P 40a763392ee34bda87d2e78be56c9d3f: Deleting consensus metadata
I20260620 09:19:36.042307 16237 catalog_manager.cc:4841] TS 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571): tablet 86b9a160a45246fb9dbe9025f58559e0 (table test-table [id=1675cae7d4db442e947bf2723f21cc4d]) successfully deleted
I20260620 09:19:36.046346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 16579
I20260620 09:19:36.050208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 16217
I20260620 09:19:56.054941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34225
--webserver_interface=127.0.98.62
--webserver_port=35923
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:19:56.145226 16741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:19:56.145509 16741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:19:56.145536 16741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:19:56.146917 16741 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 09:19:56.146974 16741 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 09:19:56.146986 16741 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 09:19:56.146997 16741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:19:56.147009 16741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:19:56.147022 16741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:19:56.147032 16741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:19:56.147044 16741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:19:56.148777 16741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:19:56.149066 16741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:19:56.149333 16741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:19:56.155568 16746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:56.155591 16747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:56.155696 16741 server_base.cc:1034] running on GCE node
W20260620 09:19:56.155568 16749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:56.156234 16741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:19:56.156621 16741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:19:56.157788 16741 hybrid_clock.cc:648] HybridClock initialized: now 1781947196157778 us; error 33 us; skew 500 ppm
Jun 20 09:19:56 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:19:56.160586 16741 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:19:56.161078 16741 webserver.cc:460] Webserver started at http://127.0.98.62:35923/ using document root <none> and password file <none>
I20260620 09:19:56.161300 16741 fs_manager.cc:359] Metadata directory not provided
I20260620 09:19:56.161350 16741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:19:56.162782 16741 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:19:56.163480 16756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:56.163697 16741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:19:56.163790 16741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "fb098963851b4bcf96304a38400a198e"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "4826fcc9aabcedb4891a0e76f2b6a828"
server_key_iv: "b8a01f65908b9d315561e5f7c2e545ad"
server_key_version: "encryptionkey@0"
I20260620 09:19:56.164191 16741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:19:56.175673 16741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:19:56.177862 16741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:19:56.178013 16741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:19:56.182363 16741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34225
I20260620 09:19:56.182427 16808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34225 every 8 connection(s)
I20260620 09:19:56.182806 16741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:19:56.184602 16809 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:19:56.185298 16809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Bootstrap starting.
I20260620 09:19:56.190775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 16741
I20260620 09:19:56.191179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41443
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34225
--webserver_port=35837
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:19:56.196589 16809 log.cc:824] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:19:56.199117 16809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:19:56.199477 16809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Bootstrap complete.
I20260620 09:19:56.201877 16809 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260620 09:19:56.202133 16809 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb098963851b4bcf96304a38400a198e, State: Initialized, Role: FOLLOWER
I20260620 09:19:56.202307 16809 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260620 09:19:56.202378 16809 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:19:56.202420 16809 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:19:56.202468 16809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:19:56.203745 16809 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260620 09:19:56.203861 16809 leader_election.cc:302] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb098963851b4bcf96304a38400a198e; no voters:
I20260620 09:19:56.204057 16809 leader_election.cc:288] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 09:19:56.204110 16813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 3 FOLLOWER]: Leader election won for term 3
I20260620 09:19:56.204241 16813 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [term 3 LEADER]: Becoming Leader. State: Replica: fb098963851b4bcf96304a38400a198e, State: Running, Role: LEADER
I20260620 09:19:56.204403 16809 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:19:56.204449 16813 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260620 09:19:56.204943 16814 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fb098963851b4bcf96304a38400a198e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } } }
I20260620 09:19:56.204962 16815 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb098963851b4bcf96304a38400a198e. Latest consensus state: current_term: 3 leader_uuid: "fb098963851b4bcf96304a38400a198e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb098963851b4bcf96304a38400a198e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } } }
I20260620 09:19:56.205125 16814 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: This master's current role is: LEADER
I20260620 09:19:56.205163 16815 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e [sys.catalog]: This master's current role is: LEADER
I20260620 09:19:56.205480 16822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:19:56.206574 16822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:19:56.206763 16822 catalog_manager.cc:1259] Loaded cluster ID: 19b5a65d18c845e8846c4ed7fbc3ef01
I20260620 09:19:56.207301 16822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:19:56.208623 16822 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:19:56.208900 16822 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Loaded TSK: 0
I20260620 09:19:56.209004 16822 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb098963851b4bcf96304a38400a198e: Loaded TSK: 1
F20260620 09:19:56.209342 16822 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa2acf9f9ad google::LogMessage::Fail() at ??:0
@ 0x7fa2acfa288a google::LogMessage::SendToLog() at ??:0
@ 0x7fa2acf9f4de google::LogMessage::Flush() at ??:0
@ 0x7fa2acfa0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa2af70c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa2af709d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa2b4ef91cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa2b4ef948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa2b4ef9594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fa2b4ebfc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa2b4ec9ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa2aee6d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa2aee62954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa2afcc66db start_thread at ??:0
@ 0x7fa2aba9d61f clone at ??:0
W20260620 09:19:56.296937 16811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:19:56.297127 16811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:19:56.297150 16811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:19:56.298283 16811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:19:56.298321 16811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:19:56.298344 16811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:19:56.299803 16811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35837
--tserver_master_addrs=127.0.98.62:34225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:19:56.300024 16811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:19:56.300230 16811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:19:56.305110 16850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:56.305226 16851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:56.305356 16811 server_base.cc:1034] running on GCE node
W20260620 09:19:56.305294 16853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:56.305651 16811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:19:56.305897 16811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:19:56.307053 16811 hybrid_clock.cc:648] HybridClock initialized: now 1781947196307034 us; error 34 us; skew 500 ppm
Jun 20 09:19:56 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:19:56.309538 16811 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:19:56.310001 16811 webserver.cc:460] Webserver started at http://127.0.98.1:35837/ using document root <none> and password file <none>
I20260620 09:19:56.310204 16811 fs_manager.cc:359] Metadata directory not provided
I20260620 09:19:56.310251 16811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:19:56.311638 16811 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:19:56.312357 16860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:56.312546 16811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:19:56.312637 16811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "23d1945502e14cc3939489b78fd55141"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "a8b2778a57d9bfa8292ae79016baccac"
server_key_iv: "a5f43373722b07195147e2a16b5e0e45"
server_key_version: "encryptionkey@0"
I20260620 09:19:56.313033 16811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:19:56.324105 16811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:19:56.326186 16811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:19:56.326329 16811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:19:56.326543 16811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:19:56.326977 16867 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:19:56.329025 16811 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:19:56.329092 16811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 09:19:56.329128 16811 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:19:56.329788 16811 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:19:56.329820 16811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 09:19:56.329913 16867 tablet_bootstrap.cc:492] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: Bootstrap starting.
I20260620 09:19:56.336475 16811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41443
I20260620 09:19:56.336539 16974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41443 every 8 connection(s)
I20260620 09:19:56.336921 16811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:19:56.337519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 16811
I20260620 09:19:56.337813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36213
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34225
--webserver_port=37533
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:19:56.340757 16867 log.cc:824] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: Log is configured to *not* fsync() on all Append() calls
I20260620 09:19:56.343649 16867 tablet_bootstrap.cc:492] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:19:56.343991 16867 tablet_bootstrap.cc:492] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: Bootstrap complete.
I20260620 09:19:56.345227 16867 ts_tablet_manager.cc:1397] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260620 09:19:56.347683 16867 raft_consensus.cc:348] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:56.348085 16867 raft_consensus.cc:729] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d1945502e14cc3939489b78fd55141, State: Initialized, Role: FOLLOWER
I20260620 09:19:56.348291 16867 consensus_queue.cc:260] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:56.348592 16867 ts_tablet_manager.cc:1428] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20260620 09:19:56.414934 16977 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:19:56.337630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:41265)
0620 09:19:56.337828 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:56.337896 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:56.338594 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:56.414736 (+ 76142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
W20260620 09:19:56.415159 16975 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34225 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34225: Client connection negotiation failed: client connection to 127.0.98.62:34225: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:19:56.415481 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:56.415356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:45025)
0620 09:19:56.415402 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:56.415458 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 09:19:56.415753 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:56.415685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:45083)
0620 09:19:56.415710 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:56.415733 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 09:19:56.415863 16975 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:19:56.423264 16978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:19:56.423488 16978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:19:56.423519 16978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:19:56.424718 16978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:19:56.424759 16978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:19:56.424795 16978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:19:56.426240 16978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37533
--tserver_master_addrs=127.0.98.62:34225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:19:56.426478 16978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:19:56.426684 16978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:19:56.431329 16986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:56.431432 16978 server_base.cc:1034] running on GCE node
W20260620 09:19:56.431313 16985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:56.431313 16988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:56.431788 16978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:19:56.431993 16978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:19:56.433125 16978 hybrid_clock.cc:648] HybridClock initialized: now 1781947196433113 us; error 30 us; skew 500 ppm
Jun 20 09:19:56 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:19:56.435468 16978 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:19:56.435954 16978 webserver.cc:460] Webserver started at http://127.0.98.2:37533/ using document root <none> and password file <none>
I20260620 09:19:56.436152 16978 fs_manager.cc:359] Metadata directory not provided
I20260620 09:19:56.436197 16978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:19:56.437415 16978 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:19:56.437995 16995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:56.438184 16978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:19:56.438263 16978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "3d0f4f7d904d45e791d014789d3b632a"
format_stamp: "Formatted at 2026-06-20 09:19:14 on dist-test-slave-rq8c"
server_key: "2ecb7289c8f999c8ed3ffce5e37111e9"
server_key_iv: "62fe539b1f7064dd70fcd39c7c03b5be"
server_key_version: "encryptionkey@0"
I20260620 09:19:56.438598 16978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:19:56.457855 16978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:19:56.459960 16978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:19:56.460074 16978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:19:56.460258 16978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:19:56.460541 16978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:19:56.460572 16978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:56.460594 16978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:19:56.460618 16978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:56.466132 16978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36213
I20260620 09:19:56.466203 17108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36213 every 8 connection(s)
I20260620 09:19:56.466518 16978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:19:56.467504 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:56.467147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:55919)
0620 09:19:56.467286 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:56.467366 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
W20260620 09:19:56.467885 17109 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34225 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34225: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
I20260620 09:19:56.468235 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:56.468107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:56063)
0620 09:19:56.468180 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:56.468212 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 09:19:56.468600 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:56.468498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:59133)
0620 09:19:56.468553 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:56.468581 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 09:19:56.468703 17109 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:19:56.473109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 16978
I20260620 09:19:56.473397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39571
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34225
--webserver_port=43101
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:19:56.553936 17112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:19:56.554147 17112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:19:56.554177 17112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:19:56.555418 17112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:19:56.555464 17112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:19:56.555500 17112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:19:56.556949 17112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43101
--tserver_master_addrs=127.0.98.62:34225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:19:56.557176 17112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:19:56.557379 17112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:19:56.562322 17117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:56.562335 17112 server_base.cc:1034] running on GCE node
W20260620 09:19:56.562302 17118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:19:56.562271 17120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:19:56.562696 17112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:19:56.562973 17112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:19:56.564145 17112 hybrid_clock.cc:648] HybridClock initialized: now 1781947196564127 us; error 32 us; skew 500 ppm
Jun 20 09:19:56 dist-test-slave-rq8c krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:19:56.566865 17112 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:19:56.567273 17112 webserver.cc:460] Webserver started at http://127.0.98.3:43101/ using document root <none> and password file <none>
I20260620 09:19:56.567463 17112 fs_manager.cc:359] Metadata directory not provided
I20260620 09:19:56.567508 17112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:19:56.568830 17112 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:19:56.569347 17127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:56.569510 17112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:19:56.569594 17112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "40a763392ee34bda87d2e78be56c9d3f"
format_stamp: "Formatted at 2026-06-20 09:19:15 on dist-test-slave-rq8c"
server_key: "388bed1bdda3bcf17969e992f866a8da"
server_key_iv: "42e9b5b6e9e7f3323786227c97f7f0a7"
server_key_version: "encryptionkey@0"
I20260620 09:19:56.569921 17112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:19:56.580637 17112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:19:56.582520 17112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:19:56.582664 17112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:19:56.582882 17112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:19:56.583149 17112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:19:56.583180 17112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:56.583209 17112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:19:56.583241 17112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:19:56.588729 17112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39571
I20260620 09:19:56.588794 17240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39571 every 8 connection(s)
I20260620 09:19:56.589093 17112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:19:56.590063 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:56.589723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:59959)
0620 09:19:56.589862 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:56.589931 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
W20260620 09:19:56.590416 17241 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34225 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34225: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
I20260620 09:19:56.590631 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:56.590556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:47281)
0620 09:19:56.590591 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:56.590615 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 09:19:56.590826 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:56.590775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:56781)
0620 09:19:56.590797 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:56.590814 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 09:19:56.590901 17241 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:19:56.598219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 17112
I20260620 09:19:57.416484 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:57.416190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:59225)
0620 09:19:57.416352 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:57.416403 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260620 09:19:57.469318 17246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:57.469061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:49751)
0620 09:19:57.469213 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:57.469258 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
I20260620 09:19:57.591425 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:57.591219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:44247)
0620 09:19:57.591337 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:57.591375 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260620 09:19:57.883491 17248 raft_consensus.cc:482] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:19:57.883652 17248 raft_consensus.cc:504] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:57.884007 17248 leader_election.cc:288] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571), 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213)
Jun 20 09:19:57 dist-test-slave-rq8c krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 09:19:57 dist-test-slave-rq8c krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 09:19:57.898713 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:57.884409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33505 (local address 127.0.98.2:36213)
0620 09:19:57.884580 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:57.884590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:57.884627 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:19:57.885078 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:57.885080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:57.885130 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:57.885256 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:57.886008 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:57.886881 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:57.888074 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:57.888482 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:57.888589 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:57.895148 (+ 6559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:57.895153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:57.895153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:57.895165 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:57.896997 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:57.897503 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:57.897507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:57.897507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:57.897610 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:57.897845 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:57.897846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:57.897846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:57.898128 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:57.898188 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:57.898473 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 09:19:57.898599 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":57,"threads_started":1}
I20260620 09:19:57.898706 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:57.884232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39571 (local address 127.0.98.1:53671)
0620 09:19:57.884290 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:57.884303 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:57.884382 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:57.885264 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:57.885267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:57.885665 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:19:57.885878 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:57.885884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:57.886950 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:57.886951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:57.887986 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:57.887989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:57.888596 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:57.888597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:57.888689 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:57.893276 (+ 4587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:19:57.893280 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:19:57.894809 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:19:57.897088 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:57.897094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:57.897096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:57.897431 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:57.897804 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:57.897805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:57.897805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:57.897868 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:57.898223 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:19:57.898224 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:19:57.898323 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 09:19:57.898411 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:19:57.898612 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":85}
I20260620 09:19:57.898710 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:57.884377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53671 (local address 127.0.98.3:39571)
0620 09:19:57.884512 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:19:57.884524 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:19:57.884559 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:19:57.885012 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:19:57.885014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:19:57.885055 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:19:57.885201 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:19:57.885990 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:57.886854 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:57.888063 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:19:57.888528 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:19:57.888621 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:57.894951 (+ 6330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:19:57.894955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:19:57.894956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:19:57.894967 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:19:57.896997 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:57.897593 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:57.897595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:57.897596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:57.897700 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:19:57.897929 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:19:57.897930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:19:57.897930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:19:57.898155 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:19:57.898231 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:19:57.898478 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 09:19:57.898598 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":51,"threads_started":1}
I20260620 09:19:57.899003 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:57.884235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36213 (local address 127.0.98.1:33505)
0620 09:19:57.884371 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:57.884381 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:19:57.884440 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:19:57.885270 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:19:57.885271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:19:57.885665 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:19:57.885878 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:57.885884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:57.887011 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:57.887012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:57.887988 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:19:57.887990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:19:57.888568 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:19:57.888569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:19:57.888687 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:19:57.893230 (+ 4543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:19:57.893239 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:19:57.895039 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:19:57.897115 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:57.897117 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:57.897117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:57.897427 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:57.897676 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:19:57.897677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:19:57.897678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:19:57.897801 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:19:57.898190 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:19:57.898197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:19:57.898324 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 09:19:57.898411 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 09:19:57.898686 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":194,"thread_start_us":66,"threads_started":1}
W20260620 09:19:57.899587 16864 leader_election.cc:341] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571): Not found: Tablet not found: 86b9a160a45246fb9dbe9025f58559e0
W20260620 09:19:57.899685 16863 leader_election.cc:341] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213): Not found: Tablet not found: 86b9a160a45246fb9dbe9025f58559e0
I20260620 09:19:57.899761 16863 leader_election.cc:302] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23d1945502e14cc3939489b78fd55141; no voters: 3d0f4f7d904d45e791d014789d3b632a, 40a763392ee34bda87d2e78be56c9d3f
I20260620 09:19:57.899859 17248 raft_consensus.cc:2726] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:19:58.417165 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:58.416899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:53419)
0620 09:19:58.417069 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:58.417119 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260620 09:19:58.470018 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:58.469780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:47625)
0620 09:19:58.469926 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:58.469972 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
I20260620 09:19:58.592149 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:58.591910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:37421)
0620 09:19:58.592056 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:58.592102 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260620 09:19:59.417884 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:59.417640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:57251)
0620 09:19:59.417787 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:59.417832 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260620 09:19:59.470825 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:59.470562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:33169)
0620 09:19:59.470731 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:59.470778 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
I20260620 09:19:59.592819 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:19:59.592603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:49891)
0620 09:19:59.592729 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 09:19:59.592773 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260620 09:19:59.758131 17258 raft_consensus.cc:482] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:19:59.758231 17258 raft_consensus.cc:504] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:19:59.758391 17258 leader_election.cc:288] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571), 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213)
W20260620 09:19:59.758818 16863 leader_election.cc:341] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213): Not found: Tablet not found: 86b9a160a45246fb9dbe9025f58559e0
W20260620 09:19:59.758901 16864 leader_election.cc:341] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571): Not found: Tablet not found: 86b9a160a45246fb9dbe9025f58559e0
I20260620 09:19:59.758956 16864 leader_election.cc:302] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23d1945502e14cc3939489b78fd55141; no voters: 3d0f4f7d904d45e791d014789d3b632a, 40a763392ee34bda87d2e78be56c9d3f
I20260620 09:19:59.759024 17258 raft_consensus.cc:2726] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:20:00.418715 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:00.418479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:40689)
0620 09:20:00.418617 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:00.418665 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260620 09:20:00.471688 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:00.471397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:57943)
0620 09:20:00.471539 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:00.471594 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260620 09:20:00.593549 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:00.593294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:46389)
0620 09:20:00.593438 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:00.593494 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260620 09:20:01.419520 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:01.419285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:52867)
0620 09:20:01.419435 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:01.419471 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260620 09:20:01.472298 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:01.472101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:40619)
0620 09:20:01.472212 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:01.472250 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260620 09:20:01.594235 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:01.594028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:46511)
0620 09:20:01.594149 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:01.594190 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260620 09:20:02.285387 17265 raft_consensus.cc:482] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:20:02.285496 17265 raft_consensus.cc:504] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:20:02.285674 17265 leader_election.cc:288] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571), 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213)
W20260620 09:20:02.286049 16864 leader_election.cc:341] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571): Not found: Tablet not found: 86b9a160a45246fb9dbe9025f58559e0
W20260620 09:20:02.286130 16863 leader_election.cc:341] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213): Not found: Tablet not found: 86b9a160a45246fb9dbe9025f58559e0
I20260620 09:20:02.286175 16863 leader_election.cc:302] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23d1945502e14cc3939489b78fd55141; no voters: 3d0f4f7d904d45e791d014789d3b632a, 40a763392ee34bda87d2e78be56c9d3f
I20260620 09:20:02.286253 17265 raft_consensus.cc:2726] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:20:02.420212 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:02.419941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:58283)
0620 09:20:02.420122 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:02.420163 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260620 09:20:02.473011 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:02.472806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:38857)
0620 09:20:02.472925 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:02.472962 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260620 09:20:02.594883 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:02.594663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:58067)
0620 09:20:02.594784 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:02.594826 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260620 09:20:03.420953 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:03.420673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:48595)
0620 09:20:03.420832 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:03.420891 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260620 09:20:03.473713 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:03.473445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:54741)
0620 09:20:03.473605 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:03.473656 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":61,"threads_started":1}
I20260620 09:20:03.595639 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:03.595398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:46785)
0620 09:20:03.595532 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:03.595574 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260620 09:20:04.421711 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:04.421470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:53103)
0620 09:20:04.421613 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:04.421659 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260620 09:20:04.474421 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:04.474185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:59013)
0620 09:20:04.474329 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:04.474374 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260620 09:20:04.596324 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:04.596081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:37029)
0620 09:20:04.596235 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:04.596277 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260620 09:20:05.422437 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:05.422208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:33777)
0620 09:20:05.422343 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:05.422389 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260620 09:20:05.475219 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:05.474966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:40315)
0620 09:20:05.475123 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:05.475169 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260620 09:20:05.596973 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:05.596773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:48085)
0620 09:20:05.596895 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:05.596929 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260620 09:20:06.379073 17278 raft_consensus.cc:482] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:20:06.379186 17278 raft_consensus.cc:504] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a763392ee34bda87d2e78be56c9d3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39571 } } peers { permanent_uuid: "3d0f4f7d904d45e791d014789d3b632a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36213 } } peers { permanent_uuid: "23d1945502e14cc3939489b78fd55141" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41443 } }
I20260620 09:20:06.379407 17278 leader_election.cc:288] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571), 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213)
W20260620 09:20:06.379745 16863 leader_election.cc:341] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d0f4f7d904d45e791d014789d3b632a (127.0.98.2:36213): Not found: Tablet not found: 86b9a160a45246fb9dbe9025f58559e0
W20260620 09:20:06.379808 16864 leader_election.cc:341] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40a763392ee34bda87d2e78be56c9d3f (127.0.98.3:39571): Not found: Tablet not found: 86b9a160a45246fb9dbe9025f58559e0
I20260620 09:20:06.379853 16864 leader_election.cc:302] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23d1945502e14cc3939489b78fd55141; no voters: 3d0f4f7d904d45e791d014789d3b632a, 40a763392ee34bda87d2e78be56c9d3f
I20260620 09:20:06.379930 17278 raft_consensus.cc:2726] T 86b9a160a45246fb9dbe9025f58559e0 P 23d1945502e14cc3939489b78fd55141 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:20:06.423214 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:06.422951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:46325)
0620 09:20:06.423111 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:06.423159 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260620 09:20:06.475944 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:06.475734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:32771)
0620 09:20:06.475854 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:06.475899 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260620 09:20:06.597780 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:06.597478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:36837)
0620 09:20:06.597658 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:06.597711 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260620 09:20:07.424026 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:07.423748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:36033)
0620 09:20:07.423912 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:07.423959 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260620 09:20:07.476810 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:07.476515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:43549)
0620 09:20:07.476687 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:07.476747 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
I20260620 09:20:07.598409 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:07.598221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:39813)
0620 09:20:07.598329 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:07.598364 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260620 09:20:08.424773 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:08.424546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:33507)
0620 09:20:08.424679 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:08.424724 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260620 09:20:08.477556 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:08.477362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:43729)
0620 09:20:08.477477 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:08.477511 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260620 09:20:08.599186 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:08.598927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:58493)
0620 09:20:08.599088 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:08.599139 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260620 09:20:09.425460 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:09.425227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:55675)
0620 09:20:09.425379 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:09.425414 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260620 09:20:09.478176 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:09.477964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:40765)
0620 09:20:09.478090 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:09.478128 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260620 09:20:09.600065 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:09.599745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:54575)
0620 09:20:09.599937 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:09.599991 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260620 09:20:10.426239 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:10.425932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:41669)
0620 09:20:10.426113 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:10.426170 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
I20260620 09:20:10.479045 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:10.478738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:54709)
0620 09:20:10.478934 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:10.478997 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260620 09:20:10.600678 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:10.600488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:58501)
0620 09:20:10.600601 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:10.600632 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260620 09:20:11.427024 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:11.426790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:57349)
0620 09:20:11.426934 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:11.426975 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260620 09:20:11.479892 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:11.479611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:34333)
0620 09:20:11.479804 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:11.479848 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20260620 09:20:11.601334 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:11.601130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:57003)
0620 09:20:11.601251 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:11.601285 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260620 09:20:11.608101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 16811
I20260620 09:20:11.612421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 16978
I20260620 09:20:11.616094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 17112
2026-06-20T09:20:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58183 ms)
[----------] 30 tests from SecurityITest (224786 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:11 dist-test-slave-rq8c krb5kdc[17299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:11 dist-test-slave-rq8c krb5kdc[17299](info): set up 2 sockets
Jun 20 09:20:11 dist-test-slave-rq8c krb5kdc[17299](info): commencing operation
krb5kdc: starting...
W20260620 09:20:12.563073 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:12 dist-test-slave-rq8c krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:20:12.669492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38273
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:12.758045 17312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:12.758257 17312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:12.758287 17312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:12.759438 17312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:12.759481 17312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:12.759500 17312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:12.759517 17312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:12.759533 17312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:12.761010 17312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:12.761252 17312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:12.761449 17312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:12.765995 17320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:12.766086 17318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:12.766018 17317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:12.766177 17312 server_base.cc:1034] running on GCE node
I20260620 09:20:12.766410 17312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:12.766640 17312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:12.767774 17312 hybrid_clock.cc:648] HybridClock initialized: now 1781947212767762 us; error 30 us; skew 500 ppm
Jun 20 09:20:12 dist-test-slave-rq8c krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:12.770004 17312 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:20:12.770421 17312 webserver.cc:460] Webserver started at http://127.0.98.62:45057/ using document root <none> and password file <none>
I20260620 09:20:12.770607 17312 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:12.770648 17312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:12.770766 17312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:12.771833 17312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "0a58ef8c200e4f948ec16f2fdc165c34"
format_stamp: "Formatted at 2026-06-20 09:20:12 on dist-test-slave-rq8c"
server_key: "58ce33f2f1c021bf8c16ed2e673e81ff"
server_key_iv: "69fff9214aae625f1005f4988d7b65c0"
server_key_version: "encryptionkey@0"
I20260620 09:20:12.772166 17312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a58ef8c200e4f948ec16f2fdc165c34"
format_stamp: "Formatted at 2026-06-20 09:20:12 on dist-test-slave-rq8c"
server_key: "58ce33f2f1c021bf8c16ed2e673e81ff"
server_key_iv: "69fff9214aae625f1005f4988d7b65c0"
server_key_version: "encryptionkey@0"
I20260620 09:20:12.773401 17312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:20:12.774117 17327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:12.774283 17312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:12.774356 17312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "0a58ef8c200e4f948ec16f2fdc165c34"
format_stamp: "Formatted at 2026-06-20 09:20:12 on dist-test-slave-rq8c"
server_key: "58ce33f2f1c021bf8c16ed2e673e81ff"
server_key_iv: "69fff9214aae625f1005f4988d7b65c0"
server_key_version: "encryptionkey@0"
I20260620 09:20:12.774421 17312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:12.789074 17312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:12.790784 17312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:12.790910 17312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:12.794865 17312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38273
I20260620 09:20:12.794909 17379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38273 every 8 connection(s)
I20260620 09:20:12.795225 17312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:12.796082 17380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:12.798246 17380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34: Bootstrap starting.
I20260620 09:20:12.798779 17380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:12.799029 17380 log.cc:824] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:12.799679 17380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34: No bootstrap required, opened a new log
I20260620 09:20:12.801190 17380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a58ef8c200e4f948ec16f2fdc165c34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38273 } }
I20260620 09:20:12.801323 17380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:12.801357 17380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a58ef8c200e4f948ec16f2fdc165c34, State: Initialized, Role: FOLLOWER
I20260620 09:20:12.801496 17380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a58ef8c200e4f948ec16f2fdc165c34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38273 } }
I20260620 09:20:12.801566 17380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:12.801602 17380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:12.801656 17380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:12.802145 17380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a58ef8c200e4f948ec16f2fdc165c34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38273 } }
I20260620 09:20:12.802251 17380 leader_election.cc:302] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a58ef8c200e4f948ec16f2fdc165c34; no voters:
I20260620 09:20:12.802403 17380 leader_election.cc:288] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:12.802429 17383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:12.802553 17383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [term 1 LEADER]: Becoming Leader. State: Replica: 0a58ef8c200e4f948ec16f2fdc165c34, State: Running, Role: LEADER
I20260620 09:20:12.802647 17380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:12.802692 17383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a58ef8c200e4f948ec16f2fdc165c34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38273 } }
I20260620 09:20:12.803061 17385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a58ef8c200e4f948ec16f2fdc165c34. Latest consensus state: current_term: 1 leader_uuid: "0a58ef8c200e4f948ec16f2fdc165c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a58ef8c200e4f948ec16f2fdc165c34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38273 } } }
I20260620 09:20:12.803201 17385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:12.803411 17392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:12.803196 17384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a58ef8c200e4f948ec16f2fdc165c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a58ef8c200e4f948ec16f2fdc165c34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38273 } } }
I20260620 09:20:12.803653 17384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:12.803786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 17312
I20260620 09:20:12.803879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:12.804049 392 external_mini_cluster.cc:1428] Setting key 72e419d8dbea0b95a63cc7044d14abd5
I20260620 09:20:12.804517 17392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:12.806460 17392 catalog_manager.cc:1347] Generated new cluster ID: 36c644c5ad5341d28fec8ad75169d606
I20260620 09:20:12.806514 17392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:20:12 dist-test-slave-rq8c krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:12.815825 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:12.804471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.0.98.62:38273)
0620 09:20:12.804778 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:12.804787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:12.807840 (+ 3053us) server_negotiation.cc:407] Connection header received
0620 09:20:12.807978 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:12.807985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:12.808028 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:12.808137 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:12.808717 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:12.809624 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:12.810346 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:12.810761 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:12.810854 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:12.812968 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:12.812972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:12.812972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:12.812983 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:12.814449 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:12.814756 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:12.814759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:12.814759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:12.814840 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:12.815003 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:12.815003 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:12.815004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:12.815232 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:12.815366 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:12.815456 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 09:20:12.815586 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":85,"threads_started":1}
I20260620 09:20:12.818658 17392 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:12.819109 17392 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:12.827104 17392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34: Generated new TSK 0
I20260620 09:20:12.827287 17392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:20:12.854266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38273
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:12.932284 17406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:12.932483 17406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:12.932507 17406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:12.933610 17406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:12.933648 17406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:12.933673 17406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:20:12.935091 17406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:12.935271 17406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:12.935446 17406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:12.940035 17412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:12.940035 17411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:12.940052 17414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:12.940318 17406 server_base.cc:1034] running on GCE node
I20260620 09:20:12.940523 17406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:12.940734 17406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:12.941869 17406 hybrid_clock.cc:648] HybridClock initialized: now 1781947212941851 us; error 35 us; skew 500 ppm
Jun 20 09:20:12 dist-test-slave-rq8c krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:12.944280 17406 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:20:12.944651 17406 webserver.cc:460] Webserver started at http://127.0.98.1:44283/ using document root <none> and password file <none>
I20260620 09:20:12.944830 17406 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:12.944868 17406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:12.944955 17406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:12.945917 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "ab48b9bfb066461ca4653ee68bcc81ac"
format_stamp: "Formatted at 2026-06-20 09:20:12 on dist-test-slave-rq8c"
server_key: "8e5191a41277a8a973c8a0cb4fe3194b"
server_key_iv: "e779c08cb1e94955db1663f8700b6bc7"
server_key_version: "encryptionkey@0"
I20260620 09:20:12.946204 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ab48b9bfb066461ca4653ee68bcc81ac"
format_stamp: "Formatted at 2026-06-20 09:20:12 on dist-test-slave-rq8c"
server_key: "8e5191a41277a8a973c8a0cb4fe3194b"
server_key_iv: "e779c08cb1e94955db1663f8700b6bc7"
server_key_version: "encryptionkey@0"
I20260620 09:20:12.947291 17406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:20:12.947919 17421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:12.948092 17406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:12.948175 17406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "ab48b9bfb066461ca4653ee68bcc81ac"
format_stamp: "Formatted at 2026-06-20 09:20:12 on dist-test-slave-rq8c"
server_key: "8e5191a41277a8a973c8a0cb4fe3194b"
server_key_iv: "e779c08cb1e94955db1663f8700b6bc7"
server_key_version: "encryptionkey@0"
I20260620 09:20:12.948230 17406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:12.970335 17406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:12.972141 17406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:12.972249 17406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:12.972430 17406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:12.972723 17406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:12.972755 17406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:12.972777 17406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:12.972800 17406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:12.978145 17406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45231
I20260620 09:20:12.978250 17534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45231 every 8 connection(s)
I20260620 09:20:12.978499 17406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:20:12 dist-test-slave-rq8c krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:12.986510 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:12.979125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51077 (local address 127.0.98.62:38273)
0620 09:20:12.979170 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:12.979171 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:12.979368 (+ 197us) server_negotiation.cc:407] Connection header received
0620 09:20:12.980250 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:12.980251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:12.980264 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:12.980311 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:12.980860 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:12.981349 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:12.982275 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:12.982458 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:12.982501 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:12.984464 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:12.984465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:12.984465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:12.984471 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:12.985522 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:12.985847 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:12.985848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:12.985848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:12.985891 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:12.986057 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:12.986057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:12.986058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:12.986149 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:12.986192 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:12.986384 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 09:20:12.986444 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:20:12.986676 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:12.979113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38273 (local address 127.0.98.1:51077)
0620 09:20:12.979244 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:12.979280 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:12.980166 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:12.980370 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:12.980373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:12.980611 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:12.980797 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:12.980801 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:12.981409 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:12.981409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:12.982218 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:12.982220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:12.982510 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:12.982511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:12.982607 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:12.983086 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:12.983095 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:12.984384 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:12.985583 (+ 1199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:12.985589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:12.985590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:12.985793 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:12.985935 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:12.985935 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:12.985935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:12.986018 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:12.986189 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:12.986191 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:12.986295 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 09:20:12.986368 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:20:12.986476 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260620 09:20:12.987144 17535 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38273
I20260620 09:20:12.987223 17535 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:12.987356 17535 heartbeater.cc:511] Master 127.0.98.62:38273 requested a full tablet report, sending...
I20260620 09:20:12.987840 17344 ts_manager.cc:194] Registered new tserver with Master: ab48b9bfb066461ca4653ee68bcc81ac (127.0.98.1:45231)
I20260620 09:20:12.988420 17344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51077
I20260620 09:20:12.988505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 17406
I20260620 09:20:12.988597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:20:12.988762 392 external_mini_cluster.cc:1428] Setting key a47bbb8e385d828359e28ae165c93361
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:20:13.008265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38273
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:13.086014 17540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:13.086199 17540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:13.086220 17540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:13.087317 17540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:13.087355 17540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:13.087379 17540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:20:13.088826 17540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:13.088999 17540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:13.089151 17540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:13.094022 17548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:13.093921 17545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:13.093911 17546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:13.094219 17540 server_base.cc:1034] running on GCE node
I20260620 09:20:13.094480 17540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:13.094664 17540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:13.095800 17540 hybrid_clock.cc:648] HybridClock initialized: now 1781947213095780 us; error 35 us; skew 500 ppm
Jun 20 09:20:13 dist-test-slave-rq8c krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:13.098804 17540 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:20:13.099292 17540 webserver.cc:460] Webserver started at http://127.0.98.2:38487/ using document root <none> and password file <none>
I20260620 09:20:13.099527 17540 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:13.099587 17540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:13.099768 17540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:13.101018 17540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "a33c92647cf54ee1a5736929fa8dabff"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "8622cb5d10299011c0e008b27248edc2"
server_key_iv: "ebbd4931df92402e802712e2c0e8717e"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.101300 17540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a33c92647cf54ee1a5736929fa8dabff"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "8622cb5d10299011c0e008b27248edc2"
server_key_iv: "ebbd4931df92402e802712e2c0e8717e"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.102450 17540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:13.103070 17555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.103250 17540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:13.103363 17540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "a33c92647cf54ee1a5736929fa8dabff"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "8622cb5d10299011c0e008b27248edc2"
server_key_iv: "ebbd4931df92402e802712e2c0e8717e"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.103433 17540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:13.112142 17540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:13.114151 17540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:13.114289 17540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:13.114490 17540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:13.114764 17540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:13.114796 17540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.114822 17540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:13.114846 17540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.120287 17540 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41681
I20260620 09:20:13.120347 17668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41681 every 8 connection(s)
I20260620 09:20:13.120633 17540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:20:13.122148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 17540
I20260620 09:20:13.122256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:20:13.122444 392 external_mini_cluster.cc:1428] Setting key ac08e1773a03ba3beaca22985862c7e8
Jun 20 09:20:13 dist-test-slave-rq8c krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:20:13.129735 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.121278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49187 (local address 127.0.98.62:38273)
0620 09:20:13.121323 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:13.121325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:13.121521 (+ 196us) server_negotiation.cc:407] Connection header received
0620 09:20:13.122530 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:13.122531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:13.122544 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:13.122595 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:13.123308 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.123836 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.124818 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.125003 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.125057 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.127240 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:13.127241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:13.127242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:13.127250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:13.128525 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:13.128872 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:13.128873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:13.128874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:13.128927 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:13.129141 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:13.129142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:13.129142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:13.129250 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:13.129304 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:13.129552 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 09:20:13.129644 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:20:13.129891 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.121259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38273 (local address 127.0.98.2:49187)
0620 09:20:13.121410 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:13.121452 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:13.122420 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:13.122677 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:13.122682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:13.122986 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:13.123226 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.123231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.123913 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.123914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.124733 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.124735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.125073 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.125074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.125200 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.125702 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:13.125710 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:13.127149 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:13.128592 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:13.128599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:13.128601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:13.128809 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:13.128979 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:13.128979 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:13.128980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:13.129080 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:13.129302 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:13.129308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:13.129440 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 09:20:13.129525 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 09:20:13.129683 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260620 09:20:13.130314 17669 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38273
I20260620 09:20:13.130401 17669 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:13.130616 17669 heartbeater.cc:511] Master 127.0.98.62:38273 requested a full tablet report, sending...
I20260620 09:20:13.131011 17344 ts_manager.cc:194] Registered new tserver with Master: a33c92647cf54ee1a5736929fa8dabff (127.0.98.2:41681)
I20260620 09:20:13.131376 17344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:20:13.143919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38273
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:13.223376 17674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:13.223603 17674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:13.223660 17674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:13.224849 17674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:13.224891 17674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:13.224931 17674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:20:13.226430 17674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:13.226671 17674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:13.226881 17674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:13.231691 17680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:13.231737 17682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:13.231726 17674 server_base.cc:1034] running on GCE node
W20260620 09:20:13.231695 17679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:13.232103 17674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:13.232335 17674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:13.233471 17674 hybrid_clock.cc:648] HybridClock initialized: now 1781947213233448 us; error 41 us; skew 500 ppm
Jun 20 09:20:13 dist-test-slave-rq8c krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:13.235937 17674 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:20:13.236375 17674 webserver.cc:460] Webserver started at http://127.0.98.3:41515/ using document root <none> and password file <none>
I20260620 09:20:13.236584 17674 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:13.236629 17674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:13.236749 17674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:13.237773 17674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "a5d83bbd0f88447d9b8e14af57a58ecd"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "6bc7bc39e733b241ee41fd3091953d9a"
server_key_iv: "e19085948953255074ffe057a901893f"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.238094 17674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a5d83bbd0f88447d9b8e14af57a58ecd"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "6bc7bc39e733b241ee41fd3091953d9a"
server_key_iv: "e19085948953255074ffe057a901893f"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.239310 17674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:20:13.240041 17689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.240181 17674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.240258 17674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "a5d83bbd0f88447d9b8e14af57a58ecd"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "6bc7bc39e733b241ee41fd3091953d9a"
server_key_iv: "e19085948953255074ffe057a901893f"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.240319 17674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:13.251403 17674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:13.253681 17674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:13.253834 17674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:13.254072 17674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:13.254522 17674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:13.254576 17674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.254608 17674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:13.254642 17674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.260318 17674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32863
I20260620 09:20:13.260370 17802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32863 every 8 connection(s)
I20260620 09:20:13.260682 17674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:20:13 dist-test-slave-rq8c krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:13.268522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 17674
I20260620 09:20:13.268612 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:20:13.268779 392 external_mini_cluster.cc:1428] Setting key 41ed9613cd19986bc46bd71abbbf17b0
I20260620 09:20:13.270138 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.261417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54633 (local address 127.0.98.62:38273)
0620 09:20:13.261502 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:13.261504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:13.261641 (+ 137us) server_negotiation.cc:407] Connection header received
0620 09:20:13.262651 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:13.262652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:13.262666 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:13.262726 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:13.263366 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.263938 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.264952 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.265148 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.265213 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.267382 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:13.267384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:13.267384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:13.267394 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:13.268736 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:13.269184 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:13.269186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:13.269186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:13.269242 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:13.269463 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:13.269464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:13.269464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:13.269582 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:13.269638 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:13.269929 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 09:20:13.270014 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 09:20:13.270267 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.261333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38273 (local address 127.0.98.3:54633)
0620 09:20:13.261513 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:13.261557 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:13.262535 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:13.262793 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:13.262797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:13.263070 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:13.263268 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.263272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.264027 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.264028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.264857 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.264860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.265226 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.265228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.265353 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.265897 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:13.265906 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:13.267279 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:13.268808 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:13.268817 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:13.268819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:13.269116 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:13.269296 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:13.269296 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:13.269297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:13.269408 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:13.269645 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:13.269652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:13.269789 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 09:20:13.269877 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:20:13.270043 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260620 09:20:13.270687 17803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38273
I20260620 09:20:13.270777 17803 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:13.270951 17803 heartbeater.cc:511] Master 127.0.98.62:38273 requested a full tablet report, sending...
I20260620 09:20:13.271287 17344 ts_manager.cc:194] Registered new tserver with Master: a5d83bbd0f88447d9b8e14af57a58ecd (127.0.98.3:32863)
I20260620 09:20:13.271659 17344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54633
I20260620 09:20:13.280622 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:20:13.288232 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.281987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.0.98.62:38273)
0620 09:20:13.282058 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:13.282061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:13.282142 (+ 81us) server_negotiation.cc:407] Connection header received
0620 09:20:13.282202 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:13.282203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:13.282213 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:13.282260 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:13.282871 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.283377 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.284415 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.284630 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.284690 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.285942 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:13.285943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:13.285943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:13.285948 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:13.287233 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:13.287543 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:13.287544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:13.287544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:13.287593 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:13.287778 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:13.287779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:13.287779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:13.287933 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:13.288014 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:13.288100 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 09:20:13.288163 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:20:13.289882 17344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:20:13.291445 17344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:20:13.298394 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.294654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.0.98.2:41681)
0620 09:20:13.294833 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:13.294837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:13.294845 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:13.294879 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:13.294881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:13.294894 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:13.295097 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:13.295175 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.296238 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.297353 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.298042 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.298147 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.298208 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:13.298265 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 09:20:13.298309 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260620 09:20:13.298383 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.294205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41681 (local address 127.0.0.1:58318)
0620 09:20:13.294690 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:13.294699 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:13.294735 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:13.294991 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:13.294992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:13.294993 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:13.295030 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.295031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.296348 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.296349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.297279 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.297281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.298150 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.298150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.298201 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.298204 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:13.298251 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:20:13.298284 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":25,"thread_start_us":77,"threads_started":1}
I20260620 09:20:13.298552 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.294311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.0.98.1:45231)
0620 09:20:13.294863 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:13.294868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:13.294877 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:20:13.294916 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:13.294918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:13.294932 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:13.295148 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:13.295214 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.295941 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.297398 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.297970 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.298032 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.298098 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:13.298428 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 09:20:13.298471 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":88,"threads_started":1}
I20260620 09:20:13.298383 17811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.294191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45231 (local address 127.0.0.1:33742)
0620 09:20:13.294465 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:13.294505 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:13.294580 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:13.295023 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:13.295024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:13.295027 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:13.295079 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.295081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.296034 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.296035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.297302 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.297305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.298049 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.298050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.298130 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.298145 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 09:20:13.298213 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 09:20:13.298275 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260620 09:20:13.298741 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.294455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.0.98.3:32863)
0620 09:20:13.294629 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:13.294634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:13.294645 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:20:13.294683 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:13.294685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:13.294700 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:13.294769 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:13.295031 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.296203 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.297551 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.298224 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.298297 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.298348 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:13.298623 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 09:20:13.298665 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260620 09:20:13.298869 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.294205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32863 (local address 127.0.0.1:45472)
0620 09:20:13.294489 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:13.294521 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:13.294580 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:13.294874 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:13.294877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:13.294879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:13.294957 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.294959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.296437 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.296438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.297480 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.297481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.298382 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.298383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.298457 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.298462 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:20:13.298509 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:20:13.298550 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":99,"threads_started":1}
I20260620 09:20:13.299597 17603 tablet_service.cc:1467] Processing CreateTablet for tablet 410efabcc72044108f4ec9d3a144b4df (DEFAULT_TABLE table=test-table [id=681ab9eea28341bb8883d841ee7cd05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:13.299597 17737 tablet_service.cc:1467] Processing CreateTablet for tablet 410efabcc72044108f4ec9d3a144b4df (DEFAULT_TABLE table=test-table [id=681ab9eea28341bb8883d841ee7cd05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:13.300030 17737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410efabcc72044108f4ec9d3a144b4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:13.300030 17603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410efabcc72044108f4ec9d3a144b4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:13.300014 17469 tablet_service.cc:1467] Processing CreateTablet for tablet 410efabcc72044108f4ec9d3a144b4df (DEFAULT_TABLE table=test-table [id=681ab9eea28341bb8883d841ee7cd05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:13.300323 17469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 410efabcc72044108f4ec9d3a144b4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:13.303190 17817 tablet_bootstrap.cc:492] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac: Bootstrap starting.
I20260620 09:20:13.303390 17819 tablet_bootstrap.cc:492] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd: Bootstrap starting.
I20260620 09:20:13.304091 17817 tablet_bootstrap.cc:654] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:13.304091 17819 tablet_bootstrap.cc:654] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:13.304682 17819 log.cc:824] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:13.304682 17817 log.cc:824] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:13.305020 17818 tablet_bootstrap.cc:492] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff: Bootstrap starting.
I20260620 09:20:13.305474 17819 tablet_bootstrap.cc:492] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd: No bootstrap required, opened a new log
I20260620 09:20:13.305557 17817 tablet_bootstrap.cc:492] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac: No bootstrap required, opened a new log
I20260620 09:20:13.305629 17817 ts_tablet_manager.cc:1397] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 09:20:13.305657 17818 tablet_bootstrap.cc:654] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:13.305555 17819 ts_tablet_manager.cc:1397] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 09:20:13.306241 17818 log.cc:824] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:13.307085 17818 tablet_bootstrap.cc:492] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff: No bootstrap required, opened a new log
I20260620 09:20:13.307153 17818 ts_tablet_manager.cc:1397] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:20:13.307402 17817 raft_consensus.cc:348] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48b9bfb066461ca4653ee68bcc81ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45231 } } peers { permanent_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } peers { permanent_uuid: "a33c92647cf54ee1a5736929fa8dabff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41681 } }
I20260620 09:20:13.307549 17817 raft_consensus.cc:374] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:13.307580 17817 raft_consensus.cc:729] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab48b9bfb066461ca4653ee68bcc81ac, State: Initialized, Role: FOLLOWER
I20260620 09:20:13.307760 17817 consensus_queue.cc:260] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48b9bfb066461ca4653ee68bcc81ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45231 } } peers { permanent_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } peers { permanent_uuid: "a33c92647cf54ee1a5736929fa8dabff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41681 } }
I20260620 09:20:13.307866 17819 raft_consensus.cc:348] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48b9bfb066461ca4653ee68bcc81ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45231 } } peers { permanent_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } peers { permanent_uuid: "a33c92647cf54ee1a5736929fa8dabff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41681 } }
I20260620 09:20:13.308013 17817 ts_tablet_manager.cc:1428] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:20:13.308040 17819 raft_consensus.cc:374] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:13.308084 17819 raft_consensus.cc:729] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d83bbd0f88447d9b8e14af57a58ecd, State: Initialized, Role: FOLLOWER
I20260620 09:20:13.308128 17535 heartbeater.cc:503] Master 127.0.98.62:38273 was elected leader, sending a full tablet report...
I20260620 09:20:13.308219 17819 consensus_queue.cc:260] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48b9bfb066461ca4653ee68bcc81ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45231 } } peers { permanent_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } peers { permanent_uuid: "a33c92647cf54ee1a5736929fa8dabff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41681 } }
I20260620 09:20:13.308473 17819 ts_tablet_manager.cc:1428] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:20:13.308607 17803 heartbeater.cc:503] Master 127.0.98.62:38273 was elected leader, sending a full tablet report...
I20260620 09:20:13.309100 17818 raft_consensus.cc:348] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48b9bfb066461ca4653ee68bcc81ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45231 } } peers { permanent_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } peers { permanent_uuid: "a33c92647cf54ee1a5736929fa8dabff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41681 } }
I20260620 09:20:13.309269 17818 raft_consensus.cc:374] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:13.309315 17818 raft_consensus.cc:729] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33c92647cf54ee1a5736929fa8dabff, State: Initialized, Role: FOLLOWER
I20260620 09:20:13.309427 17818 consensus_queue.cc:260] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48b9bfb066461ca4653ee68bcc81ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45231 } } peers { permanent_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } peers { permanent_uuid: "a33c92647cf54ee1a5736929fa8dabff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41681 } }
I20260620 09:20:13.309692 17818 ts_tablet_manager.cc:1428] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:20:13.309721 17669 heartbeater.cc:503] Master 127.0.98.62:38273 was elected leader, sending a full tablet report...
W20260620 09:20:13.371191 17670 tablet.cc:2369] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:20:13.451997 17825 raft_consensus.cc:482] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:20:13.452134 17825 raft_consensus.cc:504] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48b9bfb066461ca4653ee68bcc81ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45231 } } peers { permanent_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } peers { permanent_uuid: "a33c92647cf54ee1a5736929fa8dabff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41681 } }
I20260620 09:20:13.452513 17825 leader_election.cc:288] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab48b9bfb066461ca4653ee68bcc81ac (127.0.98.1:45231), a5d83bbd0f88447d9b8e14af57a58ecd (127.0.98.3:32863)
I20260620 09:20:13.455108 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.452814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47817 (local address 127.0.98.1:45231)
0620 09:20:13.452855 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:13.452857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:13.452879 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:20:13.452905 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:13.452906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:13.452917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:13.452958 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:13.453108 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.453564 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.454450 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.454804 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.454862 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.454873 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:13.455015 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 09:20:13.455048 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:20:13.455108 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.452702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45231 (local address 127.0.98.2:47817)
0620 09:20:13.452758 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:13.452771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:13.452836 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:13.452970 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:13.452971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:13.452972 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:13.453045 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.453046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.453637 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.453637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.454359 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.454361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.454874 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.454874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.454948 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.454951 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:13.454996 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:20:13.455034 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 09:20:13.455196 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.452827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42657 (local address 127.0.98.3:32863)
0620 09:20:13.452886 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:13.452888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:13.452961 (+ 73us) server_negotiation.cc:407] Connection header received
0620 09:20:13.453022 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:13.453023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:13.453037 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:13.453082 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:13.453212 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.453663 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.454549 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.454897 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.454952 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.454967 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:13.455095 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 09:20:13.455128 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 09:20:13.455224 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.452702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32863 (local address 127.0.98.2:42657)
0620 09:20:13.452895 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:13.452905 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:13.452968 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:13.453089 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:13.453090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:13.453091 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:13.453158 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.453159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.453725 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.453726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.454467 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.454469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.454965 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.454966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.455034 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.455037 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:13.455083 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 09:20:13.455120 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":15,"thread_start_us":92,"threads_started":1}
I20260620 09:20:13.455534 17757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410efabcc72044108f4ec9d3a144b4df" candidate_uuid: "a33c92647cf54ee1a5736929fa8dabff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd" is_pre_election: true
I20260620 09:20:13.455696 17757 raft_consensus.cc:2454] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a33c92647cf54ee1a5736929fa8dabff in term 0.
I20260620 09:20:13.455693 17489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410efabcc72044108f4ec9d3a144b4df" candidate_uuid: "a33c92647cf54ee1a5736929fa8dabff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab48b9bfb066461ca4653ee68bcc81ac" is_pre_election: true
I20260620 09:20:13.455839 17489 raft_consensus.cc:2454] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a33c92647cf54ee1a5736929fa8dabff in term 0.
I20260620 09:20:13.455909 17557 leader_election.cc:302] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a33c92647cf54ee1a5736929fa8dabff, a5d83bbd0f88447d9b8e14af57a58ecd; no voters:
I20260620 09:20:13.456036 17825 raft_consensus.cc:2781] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:20:13.456089 17825 raft_consensus.cc:482] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:20:13.456111 17825 raft_consensus.cc:3037] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:13.456789 17825 raft_consensus.cc:504] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48b9bfb066461ca4653ee68bcc81ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45231 } } peers { permanent_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } peers { permanent_uuid: "a33c92647cf54ee1a5736929fa8dabff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41681 } }
I20260620 09:20:13.456904 17825 leader_election.cc:288] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [CANDIDATE]: Term 1 election: Requested vote from peers ab48b9bfb066461ca4653ee68bcc81ac (127.0.98.1:45231), a5d83bbd0f88447d9b8e14af57a58ecd (127.0.98.3:32863)
I20260620 09:20:13.457086 17489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410efabcc72044108f4ec9d3a144b4df" candidate_uuid: "a33c92647cf54ee1a5736929fa8dabff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab48b9bfb066461ca4653ee68bcc81ac"
I20260620 09:20:13.457072 17757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "410efabcc72044108f4ec9d3a144b4df" candidate_uuid: "a33c92647cf54ee1a5736929fa8dabff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd"
I20260620 09:20:13.457155 17489 raft_consensus.cc:3037] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:13.457159 17757 raft_consensus.cc:3037] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:13.457787 17489 raft_consensus.cc:2454] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a33c92647cf54ee1a5736929fa8dabff in term 1.
I20260620 09:20:13.457890 17757 raft_consensus.cc:2454] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a33c92647cf54ee1a5736929fa8dabff in term 1.
I20260620 09:20:13.457957 17558 leader_election.cc:302] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a33c92647cf54ee1a5736929fa8dabff, ab48b9bfb066461ca4653ee68bcc81ac; no voters:
I20260620 09:20:13.458037 17825 raft_consensus.cc:2781] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:13.458168 17825 raft_consensus.cc:686] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 1 LEADER]: Becoming Leader. State: Replica: a33c92647cf54ee1a5736929fa8dabff, State: Running, Role: LEADER
I20260620 09:20:13.458268 17825 consensus_queue.cc:237] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48b9bfb066461ca4653ee68bcc81ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45231 } } peers { permanent_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } peers { permanent_uuid: "a33c92647cf54ee1a5736929fa8dabff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41681 } }
I20260620 09:20:13.459101 17343 catalog_manager.cc:5495] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff reported cstate change: term changed from 0 to 1, leader changed from <none> to a33c92647cf54ee1a5736929fa8dabff (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a33c92647cf54ee1a5736929fa8dabff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48b9bfb066461ca4653ee68bcc81ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a33c92647cf54ee1a5736929fa8dabff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41681 } health_report { overall_health: HEALTHY } } }
W20260620 09:20:13.479190 17536 tablet.cc:2369] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:20:13.479236 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.476906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.0.98.2:41681)
0620 09:20:13.476968 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:13.476970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:13.476978 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:13.477017 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:13.477018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:13.477042 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:13.477110 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:13.477229 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.477722 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.478473 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.478676 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.478730 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.478870 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:13.478978 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:13.479051 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:13.479112 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 09:20:13.479133 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:20:13.481082 17489 raft_consensus.cc:1261] T 410efabcc72044108f4ec9d3a144b4df P ab48b9bfb066461ca4653ee68bcc81ac [term 1 FOLLOWER]: Refusing update from remote peer a33c92647cf54ee1a5736929fa8dabff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:13.481086 17757 raft_consensus.cc:1261] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd [term 1 FOLLOWER]: Refusing update from remote peer a33c92647cf54ee1a5736929fa8dabff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:13.481392 17825 consensus_queue.cc:1035] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d83bbd0f88447d9b8e14af57a58ecd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:13.481531 17827 consensus_queue.cc:1035] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab48b9bfb066461ca4653ee68bcc81ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:13.484372 17831 mvcc.cc:205] Tried to move back new op lower bound from 7298855786415910912 to 7298855786325741568. Current Snapshot: MvccSnapshot[applied={T|T < 7298855786415910912}]
I20260620 09:20:13.486367 17342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:20:13.486450 17342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:20:13.487134 17342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a58ef8c200e4f948ec16f2fdc165c34: Sending DeleteTablet for 3 replicas of tablet 410efabcc72044108f4ec9d3a144b4df
I20260620 09:20:13.487425 17603 tablet_service.cc:1514] Processing DeleteTablet for tablet 410efabcc72044108f4ec9d3a144b4df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58318
I20260620 09:20:13.487433 17469 tablet_service.cc:1514] Processing DeleteTablet for tablet 410efabcc72044108f4ec9d3a144b4df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33742
I20260620 09:20:13.487453 17737 tablet_service.cc:1514] Processing DeleteTablet for tablet 410efabcc72044108f4ec9d3a144b4df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45472
I20260620 09:20:13.487680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 17406
I20260620 09:20:13.488652 17841 tablet_replica.cc:330] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff: stopping tablet replica
I20260620 09:20:13.488744 17841 raft_consensus.cc:2229] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:20:13.488849 17841 raft_consensus.cc:2258] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:13.488976 17842 tablet_replica.cc:330] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd: stopping tablet replica
I20260620 09:20:13.489048 17842 raft_consensus.cc:2229] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:20:13.489126 17842 raft_consensus.cc:2258] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:13.489423 17841 ts_tablet_manager.cc:1905] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:13.489496 17842 ts_tablet_manager.cc:1905] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:13.490890 17842 ts_tablet_manager.cc:1918] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:20:13.490945 17842 log.cc:1196] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/410efabcc72044108f4ec9d3a144b4df
I20260620 09:20:13.491159 17842 ts_tablet_manager.cc:1939] T 410efabcc72044108f4ec9d3a144b4df P a5d83bbd0f88447d9b8e14af57a58ecd: Deleting consensus metadata
I20260620 09:20:13.491497 17329 catalog_manager.cc:4841] TS a5d83bbd0f88447d9b8e14af57a58ecd (127.0.98.3:32863): tablet 410efabcc72044108f4ec9d3a144b4df (table test-table [id=681ab9eea28341bb8883d841ee7cd05a]) successfully deleted
I20260620 09:20:13.491858 17841 ts_tablet_manager.cc:1918] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:20:13.491910 17841 log.cc:1196] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/410efabcc72044108f4ec9d3a144b4df
I20260620 09:20:13.492115 17841 ts_tablet_manager.cc:1939] T 410efabcc72044108f4ec9d3a144b4df P a33c92647cf54ee1a5736929fa8dabff: Deleting consensus metadata
I20260620 09:20:13.492457 17328 catalog_manager.cc:4841] TS a33c92647cf54ee1a5736929fa8dabff (127.0.98.2:41681): tablet 410efabcc72044108f4ec9d3a144b4df (table test-table [id=681ab9eea28341bb8883d841ee7cd05a]) successfully deleted
W20260620 09:20:13.494947 17330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45231 (error 108)
I20260620 09:20:13.495066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 17540
I20260620 09:20:13.496109 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.495678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45231 (local address 127.0.0.1:33758)
0620 09:20:13.496034 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:13.496076 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334}
W20260620 09:20:13.496232 17330 catalog_manager.cc:4584] TS ab48b9bfb066461ca4653ee68bcc81ac (127.0.98.1:45231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 410efabcc72044108f4ec9d3a144b4df: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45231: connect: Connection refused (error 111)
I20260620 09:20:13.499541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 17674
I20260620 09:20:13.503701 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 17312
2026-06-20T09:20:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1887 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T09:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:13Z Disabled control of system clock
I20260620 09:20:13.528885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35095
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:20:13.607053 17847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:13.607262 17847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:13.607283 17847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:13.608505 17847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:13.608551 17847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:13.608562 17847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:13.608574 17847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:13.608594 17847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:13.609968 17847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:13.610212 17847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:13.610500 17847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:13.615257 17855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:13.615257 17853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:13.615468 17847 server_base.cc:1034] running on GCE node
W20260620 09:20:13.615252 17852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:13.615846 17847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:13.616137 17847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:13.617280 17847 hybrid_clock.cc:648] HybridClock initialized: now 1781947213617267 us; error 29 us; skew 500 ppm
I20260620 09:20:13.618451 17847 webserver.cc:460] Webserver started at http://127.0.98.62:44093/ using document root <none> and password file <none>
I20260620 09:20:13.618686 17847 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:13.618731 17847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:13.618852 17847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:13.619964 17847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "42071d5679614c63a2def011d6b3b94e"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "48dced5eab345b6be3c5480b7167fa54"
server_key_iv: "c3087a9525d9b8d1701dab2d1978fe45"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.620288 17847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "42071d5679614c63a2def011d6b3b94e"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "48dced5eab345b6be3c5480b7167fa54"
server_key_iv: "c3087a9525d9b8d1701dab2d1978fe45"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.621456 17847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 09:20:13.622118 17861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.622262 17847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:13.622334 17847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "42071d5679614c63a2def011d6b3b94e"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "48dced5eab345b6be3c5480b7167fa54"
server_key_iv: "c3087a9525d9b8d1701dab2d1978fe45"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.622402 17847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:13.636762 17847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:13.637085 17847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:13.637210 17847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:13.641036 17847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35095
I20260620 09:20:13.641074 17913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35095 every 8 connection(s)
I20260620 09:20:13.641394 17847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:13.642321 17914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:13.642799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 17847
I20260620 09:20:13.642903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:13.643087 392 external_mini_cluster.cc:1428] Setting key 62f6c774811e7141c9ef62215b4dd07e
I20260620 09:20:13.645043 17914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e: Bootstrap starting.
I20260620 09:20:13.645603 17914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:13.645900 17914 log.cc:824] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:13.646627 17914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e: No bootstrap required, opened a new log
I20260620 09:20:13.648317 17914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42071d5679614c63a2def011d6b3b94e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35095 } }
I20260620 09:20:13.648458 17914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:13.648488 17914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42071d5679614c63a2def011d6b3b94e, State: Initialized, Role: FOLLOWER
I20260620 09:20:13.648607 17914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42071d5679614c63a2def011d6b3b94e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35095 } }
I20260620 09:20:13.648672 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.643610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34952 (local address 127.0.98.62:35095)
0620 09:20:13.643787 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:13.643794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:13.643819 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:20:13.644238 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:13.644244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:13.644303 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:13.644413 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:13.644643 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.645537 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.646470 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.646927 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.647015 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.647809 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:13.647813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:13.647813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:13.647970 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:13.648004 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:13.648014 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:13.648063 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:13.648155 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 09:20:13.648220 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
I20260620 09:20:13.648679 17914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:13.648759 17914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:13.648799 17914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:13.649435 17914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42071d5679614c63a2def011d6b3b94e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35095 } }
I20260620 09:20:13.649530 17914 leader_election.cc:302] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42071d5679614c63a2def011d6b3b94e; no voters:
I20260620 09:20:13.649691 17914 leader_election.cc:288] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:13.649724 17919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:13.649860 17919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [term 1 LEADER]: Becoming Leader. State: Replica: 42071d5679614c63a2def011d6b3b94e, State: Running, Role: LEADER
I20260620 09:20:13.649950 17914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:13.649974 17919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42071d5679614c63a2def011d6b3b94e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35095 } }
I20260620 09:20:13.650323 17921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42071d5679614c63a2def011d6b3b94e. Latest consensus state: current_term: 1 leader_uuid: "42071d5679614c63a2def011d6b3b94e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42071d5679614c63a2def011d6b3b94e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35095 } } }
I20260620 09:20:13.650420 17921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:13.650319 17920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42071d5679614c63a2def011d6b3b94e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42071d5679614c63a2def011d6b3b94e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35095 } } }
I20260620 09:20:13.650484 17920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:13.650650 17927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:13.651144 17927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:13.652675 17927 catalog_manager.cc:1347] Generated new cluster ID: d0da1f0959604e6fbb8f8f1e17525d6a
I20260620 09:20:13.652729 17927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:13.663745 17927 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:13.664486 17927 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:13.667665 17927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e: Generated new TSK 0
I20260620 09:20:13.667816 17927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:13.669260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35095
--builtin_ntp_servers=127.0.98.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:20:13.754060 17938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:13.754254 17938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:13.754274 17938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:13.755402 17938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:13.755442 17938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:13.755465 17938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:20:13.756831 17938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:13.757045 17938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:13.757324 17938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:13.762948 17944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:13.763090 17946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:13.762974 17943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:13.763002 17938 server_base.cc:1034] running on GCE node
I20260620 09:20:13.763412 17938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:13.763645 17938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:13.764793 17938 hybrid_clock.cc:648] HybridClock initialized: now 1781947213764779 us; error 26 us; skew 500 ppm
I20260620 09:20:13.765980 17938 webserver.cc:460] Webserver started at http://127.0.98.1:38151/ using document root <none> and password file <none>
I20260620 09:20:13.766194 17938 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:13.766237 17938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:13.766356 17938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:13.767386 17938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "9caba1fcc64d4f5598293f1ae9461818"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "c054d1804d0bc56686cfe13610eb6ddf"
server_key_iv: "3d77f029d461e1c14b79f7ee750ef08b"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.767735 17938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9caba1fcc64d4f5598293f1ae9461818"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "c054d1804d0bc56686cfe13610eb6ddf"
server_key_iv: "3d77f029d461e1c14b79f7ee750ef08b"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.768921 17938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:20:13.769598 17952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.769739 17938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.769806 17938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "9caba1fcc64d4f5598293f1ae9461818"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "c054d1804d0bc56686cfe13610eb6ddf"
server_key_iv: "3d77f029d461e1c14b79f7ee750ef08b"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.769876 17938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:13.787992 17938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:13.788281 17938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:13.788405 17938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:13.788620 17938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:13.788940 17938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:13.788975 17938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.789003 17938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:13.789021 17938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.794206 17938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37507
I20260620 09:20:13.794267 18065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37507 every 8 connection(s)
I20260620 09:20:13.794552 17938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:20:13.799166 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.795204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36231 (local address 127.0.98.62:35095)
0620 09:20:13.795269 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:13.795271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:13.795539 (+ 268us) server_negotiation.cc:407] Connection header received
0620 09:20:13.796015 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:13.796017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:13.796028 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:13.796086 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:13.796685 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.797167 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.798083 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.798255 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.798296 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.798962 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:13.798963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:13.798964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:13.798967 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:13.798981 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:13.798985 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:13.799020 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:13.799082 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 09:20:13.799108 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 09:20:13.799338 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.795189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35095 (local address 127.0.98.1:36231)
0620 09:20:13.795410 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:13.795444 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:13.795936 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:13.796136 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:13.796139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:13.796148 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:13.796623 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.796627 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.797229 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.797230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.798017 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.798020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.798305 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.798306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.798402 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.798848 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:20:13.798848 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:13.798874 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:20:13.798895 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:13.799022 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:13.799024 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:13.799026 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:20:13.799068 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 09:20:13.799138 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260620 09:20:13.799804 18066 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35095
I20260620 09:20:13.799885 18066 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:13.800048 18066 heartbeater.cc:511] Master 127.0.98.62:35095 requested a full tablet report, sending...
I20260620 09:20:13.800491 17878 ts_manager.cc:194] Registered new tserver with Master: 9caba1fcc64d4f5598293f1ae9461818 (127.0.98.1:37507)
I20260620 09:20:13.801057 17878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36231
I20260620 09:20:13.804612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 17938
I20260620 09:20:13.804690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:20:13.804855 392 external_mini_cluster.cc:1428] Setting key ea7efbaa6721ef4cace5cb1c3ac147f5
I20260620 09:20:13.805645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35095
--builtin_ntp_servers=127.0.98.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:20:13.884264 18069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:13.884469 18069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:13.884498 18069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:13.885632 18069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:13.885669 18069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:13.885702 18069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:20:13.887017 18069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:13.887244 18069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:13.887509 18069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:13.892205 18074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:13.892194 18077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:13.892227 18069 server_base.cc:1034] running on GCE node
W20260620 09:20:13.892194 18075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:13.892540 18069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:13.892779 18069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:13.893923 18069 hybrid_clock.cc:648] HybridClock initialized: now 1781947213893907 us; error 34 us; skew 500 ppm
I20260620 09:20:13.895200 18069 webserver.cc:460] Webserver started at http://127.0.98.2:40363/ using document root <none> and password file <none>
I20260620 09:20:13.895424 18069 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:13.895468 18069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:13.895593 18069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:13.896661 18069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "3a43243767c448958040624d999a71bb"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "d64b4121db0925ee9214fb2b6a3bffd6"
server_key_iv: "b8a625c734821e571df39e4009de26cc"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.896986 18069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3a43243767c448958040624d999a71bb"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "d64b4121db0925ee9214fb2b6a3bffd6"
server_key_iv: "b8a625c734821e571df39e4009de26cc"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.898180 18069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:20:13.898833 18083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.898999 18069 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:20:13.899077 18069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "3a43243767c448958040624d999a71bb"
format_stamp: "Formatted at 2026-06-20 09:20:13 on dist-test-slave-rq8c"
server_key: "d64b4121db0925ee9214fb2b6a3bffd6"
server_key_iv: "b8a625c734821e571df39e4009de26cc"
server_key_version: "encryptionkey@0"
I20260620 09:20:13.899149 18069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:13.927392 18069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:13.927749 18069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:13.927891 18069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:13.928228 18069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:13.928638 18069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:13.928678 18069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.928709 18069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:13.928726 18069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:13.934157 18069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34441
I20260620 09:20:13.934207 18196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34441 every 8 connection(s)
I20260620 09:20:13.934523 18069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:20:13.939566 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.935176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49217 (local address 127.0.98.62:35095)
0620 09:20:13.935225 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:13.935227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:13.935443 (+ 216us) server_negotiation.cc:407] Connection header received
0620 09:20:13.936022 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:13.936024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:13.936038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:13.936088 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:13.936758 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.937252 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.938217 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.938413 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.938475 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.939292 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:13.939294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:13.939294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:13.939299 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:13.939315 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:13.939319 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:13.939365 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:13.939453 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 09:20:13.939484 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:20:13.939774 18199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:13.935157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35095 (local address 127.0.98.2:49217)
0620 09:20:13.935321 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:13.935356 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:13.935907 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:13.936155 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:13.936159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:13.936169 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:13.936663 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.936668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.937337 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.937338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.938125 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:13.938127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:13.938489 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:13.938490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:13.938626 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:13.939151 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:20:13.939151 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:13.939181 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:20:13.939204 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:13.939373 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:13.939378 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:13.939381 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:13.939445 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 09:20:13.939550 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260620 09:20:13.940099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 18069
I20260620 09:20:13.940179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:20:13.940469 392 external_mini_cluster.cc:1428] Setting key fc616b0bf1230fc4b83ed1014011d5fc
I20260620 09:20:13.940682 18197 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35095
I20260620 09:20:13.940780 18197 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:13.940968 18197 heartbeater.cc:511] Master 127.0.98.62:35095 requested a full tablet report, sending...
I20260620 09:20:13.941355 17878 ts_manager.cc:194] Registered new tserver with Master: 3a43243767c448958040624d999a71bb (127.0.98.2:34441)
I20260620 09:20:13.941401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35095
--builtin_ntp_servers=127.0.98.20:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:20:13.941748 17878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49217
W20260620 09:20:14.020764 18200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:14.020979 18200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:14.021008 18200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:14.022183 18200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:14.022225 18200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:14.022259 18200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:20:14.023672 18200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:14.023905 18200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:14.024179 18200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:14.028982 18208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:14.029134 18200 server_base.cc:1034] running on GCE node
W20260620 09:20:14.029026 18205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:14.028995 18206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:14.029500 18200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:14.029735 18200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:14.030893 18200 hybrid_clock.cc:648] HybridClock initialized: now 1781947214030873 us; error 33 us; skew 500 ppm
I20260620 09:20:14.032318 18200 webserver.cc:460] Webserver started at http://127.0.98.3:44823/ using document root <none> and password file <none>
I20260620 09:20:14.032543 18200 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:14.032596 18200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:14.032724 18200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:14.033784 18200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "9b8ddc8736f74d4397d8f848ed967911"
format_stamp: "Formatted at 2026-06-20 09:20:14 on dist-test-slave-rq8c"
server_key: "35db9baae0852c3a617ab600177d8f2d"
server_key_iv: "ad08e1e5ebb72884e73c2c26ede2a710"
server_key_version: "encryptionkey@0"
I20260620 09:20:14.034128 18200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9b8ddc8736f74d4397d8f848ed967911"
format_stamp: "Formatted at 2026-06-20 09:20:14 on dist-test-slave-rq8c"
server_key: "35db9baae0852c3a617ab600177d8f2d"
server_key_iv: "ad08e1e5ebb72884e73c2c26ede2a710"
server_key_version: "encryptionkey@0"
I20260620 09:20:14.035374 18200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:20:14.036242 18214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:14.036407 18200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 09:20:14.036489 18200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "9b8ddc8736f74d4397d8f848ed967911"
format_stamp: "Formatted at 2026-06-20 09:20:14 on dist-test-slave-rq8c"
server_key: "35db9baae0852c3a617ab600177d8f2d"
server_key_iv: "ad08e1e5ebb72884e73c2c26ede2a710"
server_key_version: "encryptionkey@0"
I20260620 09:20:14.036558 18200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:14.059742 18200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:14.060036 18200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:14.060163 18200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:14.060375 18200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:14.060686 18200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:14.060725 18200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:14.060753 18200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:14.060779 18200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:14.066670 18200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34025
I20260620 09:20:14.066717 18327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34025 every 8 connection(s)
I20260620 09:20:14.067022 18200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:20:14.071673 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.067688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58341 (local address 127.0.98.62:35095)
0620 09:20:14.067737 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:14.067739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:14.067993 (+ 254us) server_negotiation.cc:407] Connection header received
0620 09:20:14.068473 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:14.068474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:14.068484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:14.068520 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:14.069135 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.069620 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.070523 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.070706 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.070751 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.071414 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:14.071415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:14.071415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:14.071419 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:14.071432 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:14.071436 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:14.071473 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:14.071533 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 09:20:14.071569 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:20:14.071817 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.067685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35095 (local address 127.0.98.3:58341)
0620 09:20:14.067881 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:14.067916 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:14.068386 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:14.068578 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:14.068581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:14.068590 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:14.069069 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:14.069073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.069679 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.069680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:14.070464 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:14.070466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.070760 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.070761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:14.070859 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.071299 (+ 440us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:20:14.071299 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:14.071326 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:20:14.071346 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:14.071480 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:14.071482 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:14.071485 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:14.071529 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 09:20:14.071603 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260620 09:20:14.072252 18328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35095
I20260620 09:20:14.072337 18328 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:14.072501 18328 heartbeater.cc:511] Master 127.0.98.62:35095 requested a full tablet report, sending...
I20260620 09:20:14.072853 17878 ts_manager.cc:194] Registered new tserver with Master: 9b8ddc8736f74d4397d8f848ed967911 (127.0.98.3:34025)
I20260620 09:20:14.073210 17878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58341
I20260620 09:20:14.076256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 18200
I20260620 09:20:14.076331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:20:14.076520 392 external_mini_cluster.cc:1428] Setting key 1ff1b180caaf06104b509c2a3d57a507
I20260620 09:20:14.077337 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:20:14.081247 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.078626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.0.98.62:35095)
0620 09:20:14.078668 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:14.078670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:14.078782 (+ 112us) server_negotiation.cc:407] Connection header received
0620 09:20:14.078826 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:14.078826 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:14.078834 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:14.078875 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:14.078990 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.079453 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.080273 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.080456 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.080513 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.081087 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:14.081088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:14.081089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:14.081092 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:14.081105 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:14.081108 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:14.081140 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:14.081173 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 09:20:14.081192 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:20:14.082623 17878 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:20:14.084192 17878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:20:14.090176 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.086164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.0.98.2:34441)
0620 09:20:14.086481 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:14.086485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:14.086493 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:14.086521 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:14.086522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:14.086534 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:14.086601 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:14.086755 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.087900 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.089322 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.089806 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.089869 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.089958 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:14.090054 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 09:20:14.090091 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":70,"threads_started":1}
I20260620 09:20:14.090185 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.086282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37507 (local address 127.0.0.1:34476)
0620 09:20:14.086390 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:14.086400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:14.086447 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:14.086744 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:14.086745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:14.086747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:14.086790 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:14.086792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.087831 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.087832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:14.089089 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:14.089091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.089917 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.089918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:14.089984 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.089990 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:20:14.090018 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 09:20:14.090094 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":55,"thread_start_us":64,"threads_started":1}
I20260620 09:20:14.090305 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.086159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.0.98.3:34025)
0620 09:20:14.086727 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:14.086731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:14.086740 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:20:14.086773 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:14.086775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:14.086790 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:14.087121 (+ 331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:14.087180 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.088156 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.089109 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.089819 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.090080 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.090172 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:14.090196 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 09:20:14.090233 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":93,"threads_started":1}
I20260620 09:20:14.090423 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.086078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34025 (local address 127.0.0.1:59574)
0620 09:20:14.086250 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:14.086292 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:14.086356 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:14.086932 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:14.086933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:14.086935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:14.086969 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:14.086971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.088220 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.088222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:14.089041 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:14.089043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.089918 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.089918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:14.089984 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.089990 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:20:14.090018 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 09:20:14.090217 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":164,"thread_start_us":68,"threads_started":1}
I20260620 09:20:14.090488 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.086475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.0.98.1:37507)
0620 09:20:14.086608 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:14.086612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:14.086620 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:14.086655 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:14.086657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:14.086669 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:14.086736 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:14.086902 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.087754 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.089156 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.089835 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.090257 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.090352 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:14.090377 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 09:20:14.090418 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260620 09:20:14.090180 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.086081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34441 (local address 127.0.0.1:33410)
0620 09:20:14.086312 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:14.086319 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:14.086356 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:14.086611 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:14.086614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:14.086617 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:14.086677 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:14.086679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.087975 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.087976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:14.089257 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:14.089259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.089880 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.089881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:14.089957 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.089970 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 09:20:14.090047 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:20:14.090092 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260620 09:20:14.091135 18262 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed6d340b3634c9da8b6948bfde74b46 (DEFAULT_TABLE table=test-table [id=930507cd56114a0da63e781233829605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:14.091135 18000 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed6d340b3634c9da8b6948bfde74b46 (DEFAULT_TABLE table=test-table [id=930507cd56114a0da63e781233829605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:14.091321 18131 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed6d340b3634c9da8b6948bfde74b46 (DEFAULT_TABLE table=test-table [id=930507cd56114a0da63e781233829605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:14.091486 18000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ed6d340b3634c9da8b6948bfde74b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:14.091485 18262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ed6d340b3634c9da8b6948bfde74b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:14.091691 18131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ed6d340b3634c9da8b6948bfde74b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:14.094130 18342 tablet_bootstrap.cc:492] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb: Bootstrap starting.
I20260620 09:20:14.094636 18344 tablet_bootstrap.cc:492] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818: Bootstrap starting.
I20260620 09:20:14.094848 18342 tablet_bootstrap.cc:654] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:14.095177 18342 log.cc:824] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:14.095324 18344 tablet_bootstrap.cc:654] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:14.095602 18344 log.cc:824] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:14.095755 18343 tablet_bootstrap.cc:492] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911: Bootstrap starting.
I20260620 09:20:14.095952 18342 tablet_bootstrap.cc:492] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb: No bootstrap required, opened a new log
I20260620 09:20:14.096012 18342 ts_tablet_manager.cc:1397] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:20:14.096381 18343 tablet_bootstrap.cc:654] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:14.096477 18344 tablet_bootstrap.cc:492] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818: No bootstrap required, opened a new log
I20260620 09:20:14.096549 18344 ts_tablet_manager.cc:1397] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:20:14.096686 18343 log.cc:824] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:14.097252 18343 tablet_bootstrap.cc:492] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911: No bootstrap required, opened a new log
I20260620 09:20:14.097311 18343 ts_tablet_manager.cc:1397] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:20:14.098507 18342 raft_consensus.cc:348] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8ddc8736f74d4397d8f848ed967911" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34025 } } peers { permanent_uuid: "3a43243767c448958040624d999a71bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "9caba1fcc64d4f5598293f1ae9461818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37507 } }
I20260620 09:20:14.098680 18342 raft_consensus.cc:374] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:14.098713 18342 raft_consensus.cc:729] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a43243767c448958040624d999a71bb, State: Initialized, Role: FOLLOWER
I20260620 09:20:14.098858 18342 consensus_queue.cc:260] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8ddc8736f74d4397d8f848ed967911" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34025 } } peers { permanent_uuid: "3a43243767c448958040624d999a71bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "9caba1fcc64d4f5598293f1ae9461818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37507 } }
I20260620 09:20:14.098848 18344 raft_consensus.cc:348] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8ddc8736f74d4397d8f848ed967911" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34025 } } peers { permanent_uuid: "3a43243767c448958040624d999a71bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "9caba1fcc64d4f5598293f1ae9461818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37507 } }
I20260620 09:20:14.099025 18344 raft_consensus.cc:374] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:14.099073 18344 raft_consensus.cc:729] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9caba1fcc64d4f5598293f1ae9461818, State: Initialized, Role: FOLLOWER
I20260620 09:20:14.099112 18342 ts_tablet_manager.cc:1428] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:20:14.099201 18344 consensus_queue.cc:260] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8ddc8736f74d4397d8f848ed967911" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34025 } } peers { permanent_uuid: "3a43243767c448958040624d999a71bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "9caba1fcc64d4f5598293f1ae9461818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37507 } }
I20260620 09:20:14.099238 18197 heartbeater.cc:503] Master 127.0.98.62:35095 was elected leader, sending a full tablet report...
I20260620 09:20:14.099440 18344 ts_tablet_manager.cc:1428] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 09:20:14.099356 18343 raft_consensus.cc:348] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8ddc8736f74d4397d8f848ed967911" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34025 } } peers { permanent_uuid: "3a43243767c448958040624d999a71bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "9caba1fcc64d4f5598293f1ae9461818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37507 } }
I20260620 09:20:14.099488 18343 raft_consensus.cc:374] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:14.099485 18066 heartbeater.cc:503] Master 127.0.98.62:35095 was elected leader, sending a full tablet report...
I20260620 09:20:14.099610 18343 raft_consensus.cc:729] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8ddc8736f74d4397d8f848ed967911, State: Initialized, Role: FOLLOWER
I20260620 09:20:14.099774 18343 consensus_queue.cc:260] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8ddc8736f74d4397d8f848ed967911" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34025 } } peers { permanent_uuid: "3a43243767c448958040624d999a71bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "9caba1fcc64d4f5598293f1ae9461818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37507 } }
I20260620 09:20:14.100019 18343 ts_tablet_manager.cc:1428] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:20:14.100155 18328 heartbeater.cc:503] Master 127.0.98.62:35095 was elected leader, sending a full tablet report...
W20260620 09:20:14.185036 18198 tablet.cc:2369] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:20:14.217778 18349 raft_consensus.cc:482] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:20:14.217901 18349 raft_consensus.cc:504] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8ddc8736f74d4397d8f848ed967911" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34025 } } peers { permanent_uuid: "3a43243767c448958040624d999a71bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "9caba1fcc64d4f5598293f1ae9461818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37507 } }
I20260620 09:20:14.218262 18349 leader_election.cc:288] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b8ddc8736f74d4397d8f848ed967911 (127.0.98.3:34025), 3a43243767c448958040624d999a71bb (127.0.98.2:34441)
I20260620 09:20:14.221350 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.218538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43793 (local address 127.0.98.3:34025)
0620 09:20:14.218586 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:14.218588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:14.218595 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:20:14.218626 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:14.218626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:14.218634 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:14.218679 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:14.218821 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.219550 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.220529 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.221012 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.221067 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.221075 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:14.221264 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 09:20:14.221293 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:20:14.221426 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.218438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34025 (local address 127.0.98.1:43793)
0620 09:20:14.218494 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:14.218506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:14.218551 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:14.218689 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:14.218690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:14.218691 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:14.218757 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:14.218759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.219657 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.219659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:14.220433 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:14.220435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.221099 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.221100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:14.221178 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.221189 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 09:20:14.221262 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:20:14.221312 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:20:14.221695 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.218502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59493 (local address 127.0.98.2:34441)
0620 09:20:14.218565 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:14.218567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:14.218708 (+ 141us) server_negotiation.cc:407] Connection header received
0620 09:20:14.218761 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:14.218762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:14.218775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:14.218821 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:14.218985 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.219687 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.220879 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.221342 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.221418 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.221428 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:14.221582 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 09:20:14.221630 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:20:14.221715 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.218445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34441 (local address 127.0.98.1:59493)
0620 09:20:14.218637 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:14.218646 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:14.218705 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:14.218828 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:14.218829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:14.218830 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:14.218897 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:14.218899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.219748 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.219750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:14.220755 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:14.220758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.221426 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.221427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:14.221508 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.221513 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:20:14.221564 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 09:20:14.221644 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":22,"thread_start_us":119,"threads_started":1}
I20260620 09:20:14.221791 18282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed6d340b3634c9da8b6948bfde74b46" candidate_uuid: "9caba1fcc64d4f5598293f1ae9461818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b8ddc8736f74d4397d8f848ed967911" is_pre_election: true
I20260620 09:20:14.221992 18282 raft_consensus.cc:2454] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9caba1fcc64d4f5598293f1ae9461818 in term 0.
I20260620 09:20:14.222047 18151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed6d340b3634c9da8b6948bfde74b46" candidate_uuid: "9caba1fcc64d4f5598293f1ae9461818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a43243767c448958040624d999a71bb" is_pre_election: true
I20260620 09:20:14.222198 18151 raft_consensus.cc:2454] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9caba1fcc64d4f5598293f1ae9461818 in term 0.
I20260620 09:20:14.222241 17956 leader_election.cc:302] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b8ddc8736f74d4397d8f848ed967911, 9caba1fcc64d4f5598293f1ae9461818; no voters:
I20260620 09:20:14.222390 18349 raft_consensus.cc:2781] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:20:14.222445 18349 raft_consensus.cc:482] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:20:14.222463 18349 raft_consensus.cc:3037] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:14.223145 18349 raft_consensus.cc:504] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8ddc8736f74d4397d8f848ed967911" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34025 } } peers { permanent_uuid: "3a43243767c448958040624d999a71bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "9caba1fcc64d4f5598293f1ae9461818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37507 } }
I20260620 09:20:14.223356 18349 leader_election.cc:288] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [CANDIDATE]: Term 1 election: Requested vote from peers 9b8ddc8736f74d4397d8f848ed967911 (127.0.98.3:34025), 3a43243767c448958040624d999a71bb (127.0.98.2:34441)
I20260620 09:20:14.223459 18151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed6d340b3634c9da8b6948bfde74b46" candidate_uuid: "9caba1fcc64d4f5598293f1ae9461818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a43243767c448958040624d999a71bb"
I20260620 09:20:14.223536 18282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed6d340b3634c9da8b6948bfde74b46" candidate_uuid: "9caba1fcc64d4f5598293f1ae9461818" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b8ddc8736f74d4397d8f848ed967911"
I20260620 09:20:14.223547 18151 raft_consensus.cc:3037] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:14.223654 18282 raft_consensus.cc:3037] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:14.224509 18282 raft_consensus.cc:2454] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9caba1fcc64d4f5598293f1ae9461818 in term 1.
I20260620 09:20:14.224509 18151 raft_consensus.cc:2454] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9caba1fcc64d4f5598293f1ae9461818 in term 1.
I20260620 09:20:14.224711 17953 leader_election.cc:302] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a43243767c448958040624d999a71bb, 9caba1fcc64d4f5598293f1ae9461818; no voters:
I20260620 09:20:14.224871 18349 raft_consensus.cc:2781] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:14.225040 18349 raft_consensus.cc:686] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 1 LEADER]: Becoming Leader. State: Replica: 9caba1fcc64d4f5598293f1ae9461818, State: Running, Role: LEADER
I20260620 09:20:14.225123 18349 consensus_queue.cc:237] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8ddc8736f74d4397d8f848ed967911" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34025 } } peers { permanent_uuid: "3a43243767c448958040624d999a71bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } } peers { permanent_uuid: "9caba1fcc64d4f5598293f1ae9461818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37507 } }
I20260620 09:20:14.226094 17877 catalog_manager.cc:5495] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9caba1fcc64d4f5598293f1ae9461818 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9caba1fcc64d4f5598293f1ae9461818" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8ddc8736f74d4397d8f848ed967911" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a43243767c448958040624d999a71bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9caba1fcc64d4f5598293f1ae9461818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37507 } health_report { overall_health: HEALTHY } } }
I20260620 09:20:14.237475 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.235444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.0.98.1:37507)
0620 09:20:14.235495 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:14.235497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:14.235506 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:20:14.235539 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:14.235540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:14.235558 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:14.235629 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:14.235730 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.236195 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.236900 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:14.237080 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:14.237121 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:14.237212 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:14.237304 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:14.237358 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:14.237371 (+ 13us) server_negotiation.cc:299] Negotiation successful
0620 09:20:14.237387 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 09:20:14.239130 18151 raft_consensus.cc:1261] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb [term 1 FOLLOWER]: Refusing update from remote peer 9caba1fcc64d4f5598293f1ae9461818: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:14.239136 18282 raft_consensus.cc:1261] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911 [term 1 FOLLOWER]: Refusing update from remote peer 9caba1fcc64d4f5598293f1ae9461818: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:14.239454 18349 consensus_queue.cc:1035] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a43243767c448958040624d999a71bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:14.239585 18352 consensus_queue.cc:1035] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b8ddc8736f74d4397d8f848ed967911" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:14.242285 18357 mvcc.cc:205] Tried to move back new op lower bound from 7298855789521215488 to 7298855789466894336. Current Snapshot: MvccSnapshot[applied={T|T < 7298855789521215488}]
I20260620 09:20:14.242915 18356 mvcc.cc:205] Tried to move back new op lower bound from 7298855789521215488 to 7298855789466894336. Current Snapshot: MvccSnapshot[applied={T|T < 7298855789521215488 or (T in {7298855789521215488})}]
I20260620 09:20:14.244096 17877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:20:14.244175 17877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:20:14.244848 17877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42071d5679614c63a2def011d6b3b94e: Sending DeleteTablet for 3 replicas of tablet 5ed6d340b3634c9da8b6948bfde74b46
I20260620 09:20:14.245129 18262 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ed6d340b3634c9da8b6948bfde74b46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:14 UTC) from {username='slave'} at 127.0.0.1:59574
I20260620 09:20:14.245186 18000 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ed6d340b3634c9da8b6948bfde74b46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:14 UTC) from {username='slave'} at 127.0.0.1:34476
I20260620 09:20:14.245195 18131 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ed6d340b3634c9da8b6948bfde74b46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:14 UTC) from {username='slave'} at 127.0.0.1:33410
I20260620 09:20:14.245381 18367 tablet_replica.cc:330] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818: stopping tablet replica
I20260620 09:20:14.245390 18366 tablet_replica.cc:330] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb: stopping tablet replica
I20260620 09:20:14.245482 18366 raft_consensus.cc:2229] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:20:14.245512 18367 raft_consensus.cc:2229] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:20:14.245622 18367 raft_consensus.cc:2258] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:14.245662 18366 raft_consensus.cc:2258] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:14.245801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 17938
I20260620 09:20:14.246049 18367 ts_tablet_manager.cc:1905] T 5ed6d340b3634c9da8b6948bfde74b46 P 9caba1fcc64d4f5598293f1ae9461818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:14.246590 18366 ts_tablet_manager.cc:1905] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:14.247928 18366 ts_tablet_manager.cc:1918] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:20:14.247995 18366 log.cc:1196] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/5ed6d340b3634c9da8b6948bfde74b46
I20260620 09:20:14.248261 18366 ts_tablet_manager.cc:1939] T 5ed6d340b3634c9da8b6948bfde74b46 P 3a43243767c448958040624d999a71bb: Deleting consensus metadata
I20260620 09:20:14.248625 17862 catalog_manager.cc:4841] TS 3a43243767c448958040624d999a71bb (127.0.98.2:34441): tablet 5ed6d340b3634c9da8b6948bfde74b46 (table test-table [id=930507cd56114a0da63e781233829605]) successfully deleted
W20260620 09:20:14.251261 17865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37507 (error 108)
I20260620 09:20:14.251484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 18069
I20260620 09:20:14.251500 18365 tablet_replica.cc:330] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911: stopping tablet replica
I20260620 09:20:14.251595 18365 raft_consensus.cc:2229] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:20:14.251714 18365 raft_consensus.cc:2258] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:14.252216 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:14.251589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37507 (local address 127.0.0.1:34500)
0620 09:20:14.252150 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:14.252189 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541}
W20260620 09:20:14.252332 17865 catalog_manager.cc:4584] TS 9caba1fcc64d4f5598293f1ae9461818 (127.0.98.1:37507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ed6d340b3634c9da8b6948bfde74b46: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37507: connect: Connection refused (error 111)
I20260620 09:20:14.253094 18365 ts_tablet_manager.cc:1905] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:14.254305 18365 ts_tablet_manager.cc:1918] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:20:14.254359 18365 log.cc:1196] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/5ed6d340b3634c9da8b6948bfde74b46
I20260620 09:20:14.254547 18365 ts_tablet_manager.cc:1939] T 5ed6d340b3634c9da8b6948bfde74b46 P 9b8ddc8736f74d4397d8f848ed967911: Deleting consensus metadata
I20260620 09:20:14.254884 17865 catalog_manager.cc:4841] TS 9b8ddc8736f74d4397d8f848ed967911 (127.0.98.3:34025): tablet 5ed6d340b3634c9da8b6948bfde74b46 (table test-table [id=930507cd56114a0da63e781233829605]) successfully deleted
I20260620 09:20:14.256608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 18200
I20260620 09:20:14.260895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 17847
2026-06-20T09:20:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (756 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:14 dist-test-slave-rq8c krb5kdc[18371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:14 dist-test-slave-rq8c krb5kdc[18371](info): set up 2 sockets
Jun 20 09:20:14 dist-test-slave-rq8c krb5kdc[18371](info): commencing operation
krb5kdc: starting...
W20260620 09:20:15.199316 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:15 dist-test-slave-rq8c krb5kdc[18371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:20:15.307379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33735
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:15.386528 18384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:15.386736 18384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:15.386766 18384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:15.387938 18384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:15.387979 18384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:15.387997 18384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:15.388013 18384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:15.388028 18384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:15.389472 18384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:15.389698 18384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:15.389900 18384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:15.394400 18390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:15.394398 18389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:15.394433 18384 server_base.cc:1034] running on GCE node
W20260620 09:20:15.394407 18392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:15.394807 18384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:15.395090 18384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:15.396226 18384 hybrid_clock.cc:648] HybridClock initialized: now 1781947215396209 us; error 33 us; skew 500 ppm
Jun 20 09:20:15 dist-test-slave-rq8c krb5kdc[18371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:15.398546 18384 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:20:15.398993 18384 webserver.cc:460] Webserver started at http://127.0.98.62:39067/ using document root <none> and password file <none>
I20260620 09:20:15.399184 18384 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:15.399230 18384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:15.399348 18384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:15.400416 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "e0c4a039ae324db488bdde5eaa96ba34"
format_stamp: "Formatted at 2026-06-20 09:20:15 on dist-test-slave-rq8c"
server_key: "bdefb114a423252049caaafb0e2c4799"
server_key_iv: "5d25799eb560886f1195b656bfa51c4a"
server_key_version: "encryptionkey@0"
I20260620 09:20:15.400732 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "e0c4a039ae324db488bdde5eaa96ba34"
format_stamp: "Formatted at 2026-06-20 09:20:15 on dist-test-slave-rq8c"
server_key: "bdefb114a423252049caaafb0e2c4799"
server_key_iv: "5d25799eb560886f1195b656bfa51c4a"
server_key_version: "encryptionkey@0"
I20260620 09:20:15.401978 18384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:20:15.402693 18399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:15.402844 18384 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:15.402925 18384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "e0c4a039ae324db488bdde5eaa96ba34"
format_stamp: "Formatted at 2026-06-20 09:20:15 on dist-test-slave-rq8c"
server_key: "bdefb114a423252049caaafb0e2c4799"
server_key_iv: "5d25799eb560886f1195b656bfa51c4a"
server_key_version: "encryptionkey@0"
I20260620 09:20:15.402985 18384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:15.413928 18384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:15.416457 18384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:15.416599 18384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:15.420382 18384 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33735
I20260620 09:20:15.420416 18451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33735 every 8 connection(s)
I20260620 09:20:15.420727 18384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:15.421501 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:15.421705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 18384
I20260620 09:20:15.421811 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:15.421993 392 external_mini_cluster.cc:1428] Setting key 97c59b3e8e090f0a63e080d124066db3
I20260620 09:20:15.424415 18452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34: Bootstrap starting.
I20260620 09:20:15.425098 18452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:15.425370 18452 log.cc:824] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:15.426013 18452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34: No bootstrap required, opened a new log
I20260620 09:20:15.428125 18452 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c4a039ae324db488bdde5eaa96ba34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } }
I20260620 09:20:15.428279 18452 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:15.428315 18452 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c4a039ae324db488bdde5eaa96ba34, State: Initialized, Role: FOLLOWER
I20260620 09:20:15.428453 18452 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c4a039ae324db488bdde5eaa96ba34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } }
I20260620 09:20:15.428532 18452 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:15.428570 18452 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:15.428618 18452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 09:20:15 dist-test-slave-rq8c krb5kdc[18371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:15.429219 18452 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c4a039ae324db488bdde5eaa96ba34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } }
I20260620 09:20:15.429322 18452 leader_election.cc:302] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0c4a039ae324db488bdde5eaa96ba34; no voters:
I20260620 09:20:15.429490 18452 leader_election.cc:288] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:15.429817 18457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:15.429961 18457 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [term 1 LEADER]: Becoming Leader. State: Replica: e0c4a039ae324db488bdde5eaa96ba34, State: Running, Role: LEADER
I20260620 09:20:15.430059 18452 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:15.430106 18457 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c4a039ae324db488bdde5eaa96ba34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } }
I20260620 09:20:15.430461 18459 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0c4a039ae324db488bdde5eaa96ba34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c4a039ae324db488bdde5eaa96ba34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } } }
I20260620 09:20:15.430536 18459 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:15.430727 18463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:15.431236 18463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:15.431545 18458 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0c4a039ae324db488bdde5eaa96ba34. Latest consensus state: current_term: 1 leader_uuid: "e0c4a039ae324db488bdde5eaa96ba34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c4a039ae324db488bdde5eaa96ba34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } } }
I20260620 09:20:15.431810 18458 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:15.432796 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.422509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.0.98.62:33735)
0620 09:20:15.422698 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:15.422707 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:15.422741 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:20:15.423248 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:15.423257 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:15.423315 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:15.423460 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:15.424078 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.425351 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.426180 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.426772 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.426896 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.429251 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:15.429255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:15.429256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:15.429269 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:15.431223 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:15.431611 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:15.431633 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:15.431633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:15.431781 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:15.432006 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:15.432007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:15.432008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:15.432231 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:15.432375 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:15.432492 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 09:20:15.432624 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260620 09:20:15.433326 18463 catalog_manager.cc:1347] Generated new cluster ID: 41b959f1a4724e059fb94f2a1bd29fd7
I20260620 09:20:15.433386 18463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:15.455299 18463 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:15.455915 18463 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:15.463279 18463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34: Generated new TSK 0
I20260620 09:20:15.463460 18463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:20:15.494084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33735
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:15.579012 18478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:15.579236 18478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:15.579277 18478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:15.580473 18478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:15.580511 18478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:15.580545 18478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:20:15.581954 18478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:15.582182 18478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:15.582383 18478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:15.587129 18486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:15.587128 18484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:15.587179 18478 server_base.cc:1034] running on GCE node
W20260620 09:20:15.587129 18483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:15.587492 18478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:15.587740 18478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:15.588924 18478 hybrid_clock.cc:648] HybridClock initialized: now 1781947215588909 us; error 28 us; skew 500 ppm
Jun 20 09:20:15 dist-test-slave-rq8c krb5kdc[18371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:15.591209 18478 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:20:15.591635 18478 webserver.cc:460] Webserver started at http://127.0.98.1:44495/ using document root <none> and password file <none>
I20260620 09:20:15.591840 18478 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:15.591881 18478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:15.591997 18478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:15.592984 18478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "59f7992fa9984c7fb42627387312a113"
format_stamp: "Formatted at 2026-06-20 09:20:15 on dist-test-slave-rq8c"
server_key: "2e10aa3d8862c398de83b568fa713020"
server_key_iv: "3e806c1f15666aa6bbb3269757f5f72f"
server_key_version: "encryptionkey@0"
I20260620 09:20:15.593294 18478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "59f7992fa9984c7fb42627387312a113"
format_stamp: "Formatted at 2026-06-20 09:20:15 on dist-test-slave-rq8c"
server_key: "2e10aa3d8862c398de83b568fa713020"
server_key_iv: "3e806c1f15666aa6bbb3269757f5f72f"
server_key_version: "encryptionkey@0"
I20260620 09:20:15.594434 18478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:15.595016 18493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:15.595142 18478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:15.595220 18478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "59f7992fa9984c7fb42627387312a113"
format_stamp: "Formatted at 2026-06-20 09:20:15 on dist-test-slave-rq8c"
server_key: "2e10aa3d8862c398de83b568fa713020"
server_key_iv: "3e806c1f15666aa6bbb3269757f5f72f"
server_key_version: "encryptionkey@0"
I20260620 09:20:15.595278 18478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:15.622959 18478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:15.624825 18478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:15.624958 18478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:15.625164 18478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:15.625445 18478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:15.625476 18478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:15.625505 18478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:15.625522 18478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:15.630999 18478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46107
I20260620 09:20:15.631038 18606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46107 every 8 connection(s)
I20260620 09:20:15.631353 18478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:20:15 dist-test-slave-rq8c krb5kdc[18371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:15.638942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 18478
I20260620 09:20:15.639058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:20:15.639266 392 external_mini_cluster.cc:1428] Setting key 043a8017a248e9b2f4a99f42d05b1a0a
I20260620 09:20:15.639941 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.631994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54145 (local address 127.0.98.62:33735)
0620 09:20:15.632052 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:15.632054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:15.632246 (+ 192us) server_negotiation.cc:407] Connection header received
0620 09:20:15.633115 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:15.633116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:15.633125 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:15.633173 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:15.633716 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.634213 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.635163 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.635355 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.635398 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.637287 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:15.637289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:15.637289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:15.637295 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:15.638504 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:15.638866 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:15.638868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:15.638868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:15.638941 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:15.639238 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:15.639239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:15.639240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:15.639380 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:15.639446 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:15.639760 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 09:20:15.639853 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 09:20:15.640089 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.631994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33735 (local address 127.0.98.1:54145)
0620 09:20:15.632137 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:15.632171 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:15.633034 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:15.633228 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:15.633231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:15.633467 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:15.633652 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:15.633656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.634280 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.634280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:15.635100 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:15.635102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.635410 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.635411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:15.635503 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.635951 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:15.635959 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:15.637203 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:15.638583 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:15.638589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:15.638591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:15.638791 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:15.639033 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:15.639034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:15.639034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:15.639174 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:15.639439 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:15.639445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:15.639597 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 09:20:15.639713 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 09:20:15.639878 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260620 09:20:15.640542 18607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33735
I20260620 09:20:15.640617 18607 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:15.640779 18607 heartbeater.cc:511] Master 127.0.98.62:33735 requested a full tablet report, sending...
I20260620 09:20:15.641301 18416 ts_manager.cc:194] Registered new tserver with Master: 59f7992fa9984c7fb42627387312a113 (127.0.98.1:46107)
I20260620 09:20:15.641881 18416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54145
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:20:15.659193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33735
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:15.737402 18612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:15.737614 18612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:15.737639 18612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:15.738778 18612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:15.738813 18612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:15.738839 18612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:20:15.740360 18612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:15.740559 18612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:15.740733 18612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:15.745568 18620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:15.745580 18618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:15.745608 18612 server_base.cc:1034] running on GCE node
W20260620 09:20:15.745587 18617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:15.746028 18612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:15.746266 18612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:15.747403 18612 hybrid_clock.cc:648] HybridClock initialized: now 1781947215747397 us; error 29 us; skew 500 ppm
Jun 20 09:20:15 dist-test-slave-rq8c krb5kdc[18371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:15.749785 18612 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:20:15.750237 18612 webserver.cc:460] Webserver started at http://127.0.98.2:39399/ using document root <none> and password file <none>
I20260620 09:20:15.750419 18612 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:15.750460 18612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:15.750550 18612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:15.751529 18612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "dbdb7072eb414b2aac13819f51035dad"
format_stamp: "Formatted at 2026-06-20 09:20:15 on dist-test-slave-rq8c"
server_key: "9e1a72ea22c94a320490c7dc6bfa2c22"
server_key_iv: "fde5942be6bb756a1576708453240600"
server_key_version: "encryptionkey@0"
I20260620 09:20:15.751869 18612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dbdb7072eb414b2aac13819f51035dad"
format_stamp: "Formatted at 2026-06-20 09:20:15 on dist-test-slave-rq8c"
server_key: "9e1a72ea22c94a320490c7dc6bfa2c22"
server_key_iv: "fde5942be6bb756a1576708453240600"
server_key_version: "encryptionkey@0"
I20260620 09:20:15.753062 18612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:20:15.753700 18627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:15.753887 18612 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:15.753974 18612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "dbdb7072eb414b2aac13819f51035dad"
format_stamp: "Formatted at 2026-06-20 09:20:15 on dist-test-slave-rq8c"
server_key: "9e1a72ea22c94a320490c7dc6bfa2c22"
server_key_iv: "fde5942be6bb756a1576708453240600"
server_key_version: "encryptionkey@0"
I20260620 09:20:15.754046 18612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:15.761687 18612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:15.763767 18612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:15.763892 18612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:15.764089 18612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:15.764380 18612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:15.764412 18612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:15.764433 18612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:15.764456 18612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:15.769881 18612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42217
I20260620 09:20:15.769941 18740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42217 every 8 connection(s)
I20260620 09:20:15.770222 18612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:20:15.773260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 18612
I20260620 09:20:15.773368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:20:15.773541 392 external_mini_cluster.cc:1428] Setting key b43058c008e360182ebaedf641d00608
Jun 20 09:20:15 dist-test-slave-rq8c krb5kdc[18371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:15.780252 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.770910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49203 (local address 127.0.98.62:33735)
0620 09:20:15.770961 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:15.770963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:15.771134 (+ 171us) server_negotiation.cc:407] Connection header received
0620 09:20:15.772127 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:15.772128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:15.772140 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:15.772197 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:15.772787 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.773273 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.774396 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.774586 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.774638 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.777156 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:15.777157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:15.777157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:15.777164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:15.778739 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:15.779201 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:15.779201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:15.779202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:15.779274 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:15.779538 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:15.779539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:15.779539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:15.779702 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:15.779765 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:15.780095 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 09:20:15.780162 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:20:15.780471 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.770873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33735 (local address 127.0.98.2:49203)
0620 09:20:15.771023 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:15.771058 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:15.772029 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:15.772264 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:15.772267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:15.772518 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:15.772705 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:15.772709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.773352 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.773353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:15.774328 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:15.774331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.774649 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.774650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:15.774774 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.775321 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:15.775332 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:15.777044 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:15.778824 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:15.778831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:15.778833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:15.779128 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:15.779340 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:15.779341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:15.779341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:15.779477 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:15.779764 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:15.779770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:15.779957 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 09:20:15.780072 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 09:20:15.780243 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260620 09:20:15.781002 18741 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33735
I20260620 09:20:15.781100 18741 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:20:15.781288 18741 heartbeater.cc:511] Master 127.0.98.62:33735 requested a full tablet report, sending...
I20260620 09:20:15.781667 18416 ts_manager.cc:194] Registered new tserver with Master: dbdb7072eb414b2aac13819f51035dad (127.0.98.2:42217)
I20260620 09:20:15.782063 18416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:20:15.795608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33735
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:15.873294 18746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:15.873503 18746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:15.873525 18746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:15.874629 18746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:15.874662 18746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:15.874686 18746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:20:15.876151 18746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:15.876350 18746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:15.876550 18746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:15.881263 18752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:15.881286 18754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:15.881306 18751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:15.881305 18746 server_base.cc:1034] running on GCE node
I20260620 09:20:15.881671 18746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:15.881949 18746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:15.883103 18746 hybrid_clock.cc:648] HybridClock initialized: now 1781947215883087 us; error 29 us; skew 500 ppm
Jun 20 09:20:15 dist-test-slave-rq8c krb5kdc[18371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:15.885535 18746 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:20:15.885938 18746 webserver.cc:460] Webserver started at http://127.0.98.3:37735/ using document root <none> and password file <none>
I20260620 09:20:15.886147 18746 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:15.886193 18746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:15.886317 18746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:15.887317 18746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10"
format_stamp: "Formatted at 2026-06-20 09:20:15 on dist-test-slave-rq8c"
server_key: "1e47e341dbfc845af78cc61790f69303"
server_key_iv: "88278a62fa94c608e363c1336e2c622f"
server_key_version: "encryptionkey@0"
I20260620 09:20:15.887691 18746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10"
format_stamp: "Formatted at 2026-06-20 09:20:15 on dist-test-slave-rq8c"
server_key: "1e47e341dbfc845af78cc61790f69303"
server_key_iv: "88278a62fa94c608e363c1336e2c622f"
server_key_version: "encryptionkey@0"
I20260620 09:20:15.888943 18746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:20:15.889647 18761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:15.889796 18746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:15.889866 18746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10"
format_stamp: "Formatted at 2026-06-20 09:20:15 on dist-test-slave-rq8c"
server_key: "1e47e341dbfc845af78cc61790f69303"
server_key_iv: "88278a62fa94c608e363c1336e2c622f"
server_key_version: "encryptionkey@0"
I20260620 09:20:15.889935 18746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:15.904807 18746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:15.907063 18746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:15.907209 18746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:15.907444 18746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:15.907783 18746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:15.907819 18746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:15.907850 18746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:15.907873 18746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:15.913303 18746 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38963
I20260620 09:20:15.913347 18874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38963 every 8 connection(s)
I20260620 09:20:15.913666 18746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:20:15 dist-test-slave-rq8c krb5kdc[18371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:15.920135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 18746
I20260620 09:20:15.920264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:20:15.920483 392 external_mini_cluster.cc:1428] Setting key 346dc96bf1d6ae70dda6ec3dbadcb929
I20260620 09:20:15.922894 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.914349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53097 (local address 127.0.98.62:33735)
0620 09:20:15.914406 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:15.914418 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:15.914609 (+ 191us) server_negotiation.cc:407] Connection header received
0620 09:20:15.915521 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:15.915522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:15.915532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:15.915593 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:15.916228 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.916727 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.917882 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.918058 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.918101 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.920181 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:15.920183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:15.920183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:15.920191 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:15.921603 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:15.921994 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:15.921995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:15.921996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:15.922057 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:15.922271 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:15.922272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:15.922272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:15.922391 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:15.922445 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:15.922716 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 09:20:15.922800 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:20:15.923082 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.914313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33735 (local address 127.0.98.3:53097)
0620 09:20:15.914487 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:15.914529 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:15.915428 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:15.915685 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:15.915689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:15.915964 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:15.916159 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:15.916164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.916824 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.916826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:15.917795 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:15.917802 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.918110 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.918111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:15.918207 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.918687 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:15.918695 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:15.920062 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:15.921683 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:15.921691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:15.921694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:15.921918 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:15.922109 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:15.922109 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:15.922110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:15.922217 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:15.922451 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:15.922459 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:15.922599 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 09:20:15.922687 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:20:15.922856 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260620 09:20:15.923538 18875 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33735
I20260620 09:20:15.923646 18875 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:15.923823 18875 heartbeater.cc:511] Master 127.0.98.62:33735 requested a full tablet report, sending...
I20260620 09:20:15.924201 18416 ts_manager.cc:194] Registered new tserver with Master: 5c15aa8e3f6449f4b8ad39c59e8fea10 (127.0.98.3:38963)
I20260620 09:20:15.924623 18416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53097
I20260620 09:20:15.932518 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:20:15.939860 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.933922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39754 (local address 127.0.98.62:33735)
0620 09:20:15.933975 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:15.933976 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:15.934065 (+ 89us) server_negotiation.cc:407] Connection header received
0620 09:20:15.934113 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:15.934113 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:15.934122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:15.934168 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:15.934685 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.935170 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.935927 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.936121 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.936184 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.937529 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:15.937530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:15.937530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:15.937537 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:15.938799 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:15.939142 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:15.939143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:15.939143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:15.939194 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:15.939356 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:15.939357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:15.939357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:15.939481 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:15.939552 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:15.939685 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 09:20:15.939767 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:20:15.941448 18416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:20:15.943004 18416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:20:15.949110 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.945208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.0.98.1:46107)
0620 09:20:15.945388 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:15.945393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:15.945404 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:20:15.945471 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:15.945473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:15.945490 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:15.945562 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:15.945825 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.946616 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.947972 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.948697 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.948769 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.948826 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:15.948981 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 09:20:15.949020 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":74,"threads_started":1}
I20260620 09:20:15.949126 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.945204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38963 (local address 127.0.0.1:52098)
0620 09:20:15.945410 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:15.945419 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:15.945456 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:15.945813 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:15.945814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:15.945817 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:15.945884 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:15.945886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.947131 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.947132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:15.948192 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:15.948194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.948920 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.948921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:15.948994 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.948998 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:20:15.949038 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 09:20:15.949069 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":39,"threads_started":1}
I20260620 09:20:15.949290 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.945424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.0.98.3:38963)
0620 09:20:15.945596 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:15.945599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:15.945607 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:15.945638 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:15.945640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:15.945652 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:15.945717 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:15.945963 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.947039 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.948271 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.948747 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.948828 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.948889 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:15.949166 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 09:20:15.949210 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260620 09:20:15.949124 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.945018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.0.1:47710)
0620 09:20:15.945319 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:15.945330 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:15.945396 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:15.945663 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:15.945666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:15.945668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:15.945743 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:15.945746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.946705 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.946706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:15.947878 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:15.947881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.948784 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.948785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:15.948870 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.948883 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 09:20:15.948969 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:20:15.949032 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":52,"threads_started":1}
I20260620 09:20:15.950093 18809 tablet_service.cc:1467] Processing CreateTablet for tablet 3b52ea241a38455fb55fe87351c270c6 (DEFAULT_TABLE table=test-table [id=7b0b9b814c314a40a27bc623f20880a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:15.950135 18541 tablet_service.cc:1467] Processing CreateTablet for tablet 3b52ea241a38455fb55fe87351c270c6 (DEFAULT_TABLE table=test-table [id=7b0b9b814c314a40a27bc623f20880a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:15.950461 18809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b52ea241a38455fb55fe87351c270c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:15.950500 18541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b52ea241a38455fb55fe87351c270c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:15.950567 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.945016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42217 (local address 127.0.0.1:33836)
0620 09:20:15.945178 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:15.945219 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:15.945288 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:15.945719 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:15.945720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:15.945722 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:15.945755 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:15.945756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.948273 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.948274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:15.949274 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:15.949277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.950224 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.950225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:15.950354 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.950359 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:20:15.950414 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 09:20:15.950454 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20260620 09:20:15.950598 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:15.945156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.0.98.2:42217)
0620 09:20:15.945566 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:15.945571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:15.945581 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:20:15.945621 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:15.945622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:15.945637 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:15.945715 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:15.945880 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.947016 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.949364 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:15.950087 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:15.950159 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:15.950217 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:15.950431 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 09:20:15.950472 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":74,"threads_started":1}
I20260620 09:20:15.952106 18675 tablet_service.cc:1467] Processing CreateTablet for tablet 3b52ea241a38455fb55fe87351c270c6 (DEFAULT_TABLE table=test-table [id=7b0b9b814c314a40a27bc623f20880a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:15.952433 18675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b52ea241a38455fb55fe87351c270c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:15.953135 18889 tablet_bootstrap.cc:492] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10: Bootstrap starting.
I20260620 09:20:15.953917 18889 tablet_bootstrap.cc:654] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:15.954401 18889 log.cc:824] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:15.954711 18890 tablet_bootstrap.cc:492] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113: Bootstrap starting.
I20260620 09:20:15.955068 18889 tablet_bootstrap.cc:492] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10: No bootstrap required, opened a new log
I20260620 09:20:15.955132 18889 ts_tablet_manager.cc:1397] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:20:15.955137 18892 tablet_bootstrap.cc:492] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad: Bootstrap starting.
I20260620 09:20:15.955756 18892 tablet_bootstrap.cc:654] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:15.956167 18890 tablet_bootstrap.cc:654] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:15.956395 18892 log.cc:824] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:15.956677 18890 log.cc:824] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:15.957132 18892 tablet_bootstrap.cc:492] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad: No bootstrap required, opened a new log
I20260620 09:20:15.957206 18892 ts_tablet_manager.cc:1397] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 09:20:15.957140 18889 raft_consensus.cc:348] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7992fa9984c7fb42627387312a113" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "dbdb7072eb414b2aac13819f51035dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42217 } } peers { permanent_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38963 } }
I20260620 09:20:15.957324 18889 raft_consensus.cc:374] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:15.957362 18889 raft_consensus.cc:729] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c15aa8e3f6449f4b8ad39c59e8fea10, State: Initialized, Role: FOLLOWER
I20260620 09:20:15.957382 18890 tablet_bootstrap.cc:492] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113: No bootstrap required, opened a new log
I20260620 09:20:15.957442 18890 ts_tablet_manager.cc:1397] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:20:15.957511 18889 consensus_queue.cc:260] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7992fa9984c7fb42627387312a113" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "dbdb7072eb414b2aac13819f51035dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42217 } } peers { permanent_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38963 } }
I20260620 09:20:15.957773 18889 ts_tablet_manager.cc:1428] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:20:15.959153 18892 raft_consensus.cc:348] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7992fa9984c7fb42627387312a113" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "dbdb7072eb414b2aac13819f51035dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42217 } } peers { permanent_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38963 } }
I20260620 09:20:15.959352 18892 raft_consensus.cc:374] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:15.959389 18892 raft_consensus.cc:729] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdb7072eb414b2aac13819f51035dad, State: Initialized, Role: FOLLOWER
I20260620 09:20:15.959282 18890 raft_consensus.cc:348] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7992fa9984c7fb42627387312a113" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "dbdb7072eb414b2aac13819f51035dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42217 } } peers { permanent_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38963 } }
I20260620 09:20:15.959429 18890 raft_consensus.cc:374] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:15.959466 18890 raft_consensus.cc:729] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7992fa9984c7fb42627387312a113, State: Initialized, Role: FOLLOWER
I20260620 09:20:15.959506 18892 consensus_queue.cc:260] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7992fa9984c7fb42627387312a113" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "dbdb7072eb414b2aac13819f51035dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42217 } } peers { permanent_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38963 } }
I20260620 09:20:15.959584 18875 heartbeater.cc:503] Master 127.0.98.62:33735 was elected leader, sending a full tablet report...
I20260620 09:20:15.959590 18890 consensus_queue.cc:260] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7992fa9984c7fb42627387312a113" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "dbdb7072eb414b2aac13819f51035dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42217 } } peers { permanent_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38963 } }
I20260620 09:20:15.959808 18892 ts_tablet_manager.cc:1428] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 09:20:15.959854 18890 ts_tablet_manager.cc:1428] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 09:20:15.959844 18741 heartbeater.cc:503] Master 127.0.98.62:33735 was elected leader, sending a full tablet report...
I20260620 09:20:15.960085 18607 heartbeater.cc:503] Master 127.0.98.62:33735 was elected leader, sending a full tablet report...
W20260620 09:20:16.020753 18742 tablet.cc:2369] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:20:16.068137 18895 raft_consensus.cc:482] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:20:16.068284 18895 raft_consensus.cc:504] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7992fa9984c7fb42627387312a113" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "dbdb7072eb414b2aac13819f51035dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42217 } } peers { permanent_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38963 } }
I20260620 09:20:16.068652 18895 leader_election.cc:288] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7992fa9984c7fb42627387312a113 (127.0.98.1:46107), dbdb7072eb414b2aac13819f51035dad (127.0.98.2:42217)
I20260620 09:20:16.072484 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.068973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59511 (local address 127.0.98.1:46107)
0620 09:20:16.069017 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.069019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.069147 (+ 128us) server_negotiation.cc:407] Connection header received
0620 09:20:16.069274 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.069275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.069289 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.069365 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:16.069540 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.070217 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.071586 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.072143 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.072209 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.072222 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.072368 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.072409 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:20:16.072484 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.068882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.98.3:59511)
0620 09:20:16.069030 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:16.069042 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:16.069146 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:16.069396 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:16.069396 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:16.069398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:16.069473 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.069475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.070328 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.070330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.071481 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.071484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.072222 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.072223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.072302 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.072305 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:16.072354 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 09:20:16.072405 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 09:20:16.072832 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.069120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36179 (local address 127.0.98.2:42217)
0620 09:20:16.069161 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.069163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.069274 (+ 111us) server_negotiation.cc:407] Connection header received
0620 09:20:16.069370 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.069371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.069385 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.069469 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:16.069696 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.070432 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.071855 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.072410 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.072453 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.072462 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.072724 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.072772 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:20:16.072840 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.068882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42217 (local address 127.0.98.3:36179)
0620 09:20:16.069214 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:16.069223 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:16.069275 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:16.069485 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:16.069486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:16.069487 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:16.069580 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.069582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.070546 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.070547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.071742 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.071745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.072529 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.072530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.072603 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.072619 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 09:20:16.072708 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 09:20:16.072748 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":95,"thread_start_us":117,"threads_started":1}
I20260620 09:20:16.072829 18561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b52ea241a38455fb55fe87351c270c6" candidate_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7992fa9984c7fb42627387312a113" is_pre_election: true
I20260620 09:20:16.072999 18561 raft_consensus.cc:2454] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c15aa8e3f6449f4b8ad39c59e8fea10 in term 0.
I20260620 09:20:16.073191 18695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b52ea241a38455fb55fe87351c270c6" candidate_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbdb7072eb414b2aac13819f51035dad" is_pre_election: true
I20260620 09:20:16.073246 18762 leader_election.cc:302] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7992fa9984c7fb42627387312a113, 5c15aa8e3f6449f4b8ad39c59e8fea10; no voters:
I20260620 09:20:16.073328 18695 raft_consensus.cc:2454] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c15aa8e3f6449f4b8ad39c59e8fea10 in term 0.
I20260620 09:20:16.073374 18895 raft_consensus.cc:2781] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:20:16.073468 18895 raft_consensus.cc:482] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:20:16.073499 18895 raft_consensus.cc:3037] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:16.074229 18895 raft_consensus.cc:504] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7992fa9984c7fb42627387312a113" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "dbdb7072eb414b2aac13819f51035dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42217 } } peers { permanent_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38963 } }
I20260620 09:20:16.074365 18895 leader_election.cc:288] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [CANDIDATE]: Term 1 election: Requested vote from peers 59f7992fa9984c7fb42627387312a113 (127.0.98.1:46107), dbdb7072eb414b2aac13819f51035dad (127.0.98.2:42217)
I20260620 09:20:16.074539 18695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b52ea241a38455fb55fe87351c270c6" candidate_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbdb7072eb414b2aac13819f51035dad"
I20260620 09:20:16.074574 18561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b52ea241a38455fb55fe87351c270c6" candidate_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7992fa9984c7fb42627387312a113"
I20260620 09:20:16.074615 18695 raft_consensus.cc:3037] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:16.074657 18561 raft_consensus.cc:3037] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:16.075284 18695 raft_consensus.cc:2454] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c15aa8e3f6449f4b8ad39c59e8fea10 in term 1.
I20260620 09:20:16.075352 18561 raft_consensus.cc:2454] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c15aa8e3f6449f4b8ad39c59e8fea10 in term 1.
I20260620 09:20:16.075459 18765 leader_election.cc:302] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c15aa8e3f6449f4b8ad39c59e8fea10, dbdb7072eb414b2aac13819f51035dad; no voters:
I20260620 09:20:16.075546 18895 raft_consensus.cc:2781] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:16.075747 18895 raft_consensus.cc:686] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 1 LEADER]: Becoming Leader. State: Replica: 5c15aa8e3f6449f4b8ad39c59e8fea10, State: Running, Role: LEADER
I20260620 09:20:16.075837 18895 consensus_queue.cc:237] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7992fa9984c7fb42627387312a113" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "dbdb7072eb414b2aac13819f51035dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42217 } } peers { permanent_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38963 } }
I20260620 09:20:16.076671 18416 catalog_manager.cc:5495] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c15aa8e3f6449f4b8ad39c59e8fea10 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7992fa9984c7fb42627387312a113" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbdb7072eb414b2aac13819f51035dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c15aa8e3f6449f4b8ad39c59e8fea10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38963 } health_report { overall_health: HEALTHY } } }
I20260620 09:20:16.097604 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.095340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.0.98.3:38963)
0620 09:20:16.095388 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.095390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.095399 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:20:16.095435 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.095436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.095453 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.095514 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:16.095606 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.096115 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.096841 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.097045 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.097107 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.097217 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:16.097329 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:16.097454 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.097495 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.097503 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 09:20:16.099475 18561 raft_consensus.cc:1261] T 3b52ea241a38455fb55fe87351c270c6 P 59f7992fa9984c7fb42627387312a113 [term 1 FOLLOWER]: Refusing update from remote peer 5c15aa8e3f6449f4b8ad39c59e8fea10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:16.099475 18695 raft_consensus.cc:1261] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad [term 1 FOLLOWER]: Refusing update from remote peer 5c15aa8e3f6449f4b8ad39c59e8fea10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:16.099843 18899 consensus_queue.cc:1035] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbdb7072eb414b2aac13819f51035dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:16.100003 18895 consensus_queue.cc:1035] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7992fa9984c7fb42627387312a113" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:16.104473 18415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:20:16.104560 18415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:20:16.105271 18415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0c4a039ae324db488bdde5eaa96ba34: Sending DeleteTablet for 3 replicas of tablet 3b52ea241a38455fb55fe87351c270c6
I20260620 09:20:16.105589 18809 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b52ea241a38455fb55fe87351c270c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52098
I20260620 09:20:16.105607 18675 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b52ea241a38455fb55fe87351c270c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33836
I20260620 09:20:16.105656 18541 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b52ea241a38455fb55fe87351c270c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47710
I20260620 09:20:16.105803 18912 tablet_replica.cc:330] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10: stopping tablet replica
I20260620 09:20:16.105894 18912 raft_consensus.cc:2229] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:20:16.105975 18912 raft_consensus.cc:2258] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:16.106220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 18478
I20260620 09:20:16.106318 18912 ts_tablet_manager.cc:1905] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:16.107791 18912 ts_tablet_manager.cc:1918] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:20:16.107854 18912 log.cc:1196] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/3b52ea241a38455fb55fe87351c270c6
I20260620 09:20:16.108081 18912 ts_tablet_manager.cc:1939] T 3b52ea241a38455fb55fe87351c270c6 P 5c15aa8e3f6449f4b8ad39c59e8fea10: Deleting consensus metadata
I20260620 09:20:16.110229 18403 catalog_manager.cc:4841] TS 5c15aa8e3f6449f4b8ad39c59e8fea10 (127.0.98.3:38963): tablet 3b52ea241a38455fb55fe87351c270c6 (table test-table [id=7b0b9b814c314a40a27bc623f20880a8]) successfully deleted
I20260620 09:20:16.110488 18913 tablet_replica.cc:330] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad: stopping tablet replica
I20260620 09:20:16.110620 18913 raft_consensus.cc:2229] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:20:16.110739 18913 raft_consensus.cc:2258] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:16.111179 18913 ts_tablet_manager.cc:1905] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:16.112855 18913 ts_tablet_manager.cc:1918] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:20:16.112939 18913 log.cc:1196] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/3b52ea241a38455fb55fe87351c270c6
I20260620 09:20:16.113211 18913 ts_tablet_manager.cc:1939] T 3b52ea241a38455fb55fe87351c270c6 P dbdb7072eb414b2aac13819f51035dad: Deleting consensus metadata
I20260620 09:20:16.113664 18403 catalog_manager.cc:4841] TS dbdb7072eb414b2aac13819f51035dad (127.0.98.2:42217): tablet 3b52ea241a38455fb55fe87351c270c6 (table test-table [id=7b0b9b814c314a40a27bc623f20880a8]) successfully deleted
W20260620 09:20:16.114403 18400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46107 (error 108)
I20260620 09:20:16.114871 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.114752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.0.1:47712)
0620 09:20:16.114803 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:16.114840 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 09:20:16.114881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 18612
W20260620 09:20:16.118069 18400 catalog_manager.cc:4584] TS 59f7992fa9984c7fb42627387312a113 (127.0.98.1:46107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b52ea241a38455fb55fe87351c270c6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46107: connect: Connection refused (error 111)
I20260620 09:20:16.120977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 18746
I20260620 09:20:16.125147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 18384
2026-06-20T09:20:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1865 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T09:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:16Z Disabled control of system clock
I20260620 09:20:16.151924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41415
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:20:16.229766 18919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:16.230021 18919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:16.230044 18919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:16.231163 18919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:16.231197 18919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:16.231209 18919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:16.231220 18919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:16.231231 18919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:16.232556 18919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41415
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:16.232749 18919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:16.232995 18919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:16.237520 18927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:16.237548 18925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:16.237629 18919 server_base.cc:1034] running on GCE node
W20260620 09:20:16.237703 18924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:16.237934 18919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:16.238190 18919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:16.239333 18919 hybrid_clock.cc:648] HybridClock initialized: now 1781947216239321 us; error 25 us; skew 500 ppm
I20260620 09:20:16.240541 18919 webserver.cc:460] Webserver started at http://127.0.98.62:37991/ using document root <none> and password file <none>
I20260620 09:20:16.240751 18919 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:16.240798 18919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:16.240921 18919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:16.241933 18919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "61c98f008c6d416ba115309b93b4b55b"
format_stamp: "Formatted at 2026-06-20 09:20:16 on dist-test-slave-rq8c"
server_key: "63241e5499382815b72898643944c183"
server_key_iv: "1893d0b5db9d2a04bd944c5e8da1af95"
server_key_version: "encryptionkey@0"
I20260620 09:20:16.242269 18919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "61c98f008c6d416ba115309b93b4b55b"
format_stamp: "Formatted at 2026-06-20 09:20:16 on dist-test-slave-rq8c"
server_key: "63241e5499382815b72898643944c183"
server_key_iv: "1893d0b5db9d2a04bd944c5e8da1af95"
server_key_version: "encryptionkey@0"
I20260620 09:20:16.243456 18919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:16.244204 18933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.244374 18919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.244459 18919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "61c98f008c6d416ba115309b93b4b55b"
format_stamp: "Formatted at 2026-06-20 09:20:16 on dist-test-slave-rq8c"
server_key: "63241e5499382815b72898643944c183"
server_key_iv: "1893d0b5db9d2a04bd944c5e8da1af95"
server_key_version: "encryptionkey@0"
I20260620 09:20:16.244524 18919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:16.282693 18919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:16.283048 18919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:16.283182 18919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:16.287009 18919 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41415
I20260620 09:20:16.287048 18985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41415 every 8 connection(s)
I20260620 09:20:16.287379 18919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:16.288335 18986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:16.290572 18986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b: Bootstrap starting.
I20260620 09:20:16.291129 18986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:16.291397 18986 log.cc:824] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:16.292061 18986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b: No bootstrap required, opened a new log
I20260620 09:20:16.293602 18986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c98f008c6d416ba115309b93b4b55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } }
I20260620 09:20:16.293740 18986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:16.293774 18986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c98f008c6d416ba115309b93b4b55b, State: Initialized, Role: FOLLOWER
I20260620 09:20:16.293906 18986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c98f008c6d416ba115309b93b4b55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } }
I20260620 09:20:16.293982 18986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:16.294008 18986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:16.294052 18986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:16.294550 18986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c98f008c6d416ba115309b93b4b55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } }
I20260620 09:20:16.294636 18986 leader_election.cc:302] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c98f008c6d416ba115309b93b4b55b; no voters:
I20260620 09:20:16.294821 18986 leader_election.cc:288] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:16.294863 18989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:16.295029 18989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [term 1 LEADER]: Becoming Leader. State: Replica: 61c98f008c6d416ba115309b93b4b55b, State: Running, Role: LEADER
I20260620 09:20:16.295068 18986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:16.295130 18989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c98f008c6d416ba115309b93b4b55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } }
I20260620 09:20:16.295606 18990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61c98f008c6d416ba115309b93b4b55b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c98f008c6d416ba115309b93b4b55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } }
I20260620 09:20:16.295704 18990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:16.295794 18991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61c98f008c6d416ba115309b93b4b55b. Latest consensus state: current_term: 1 leader_uuid: "61c98f008c6d416ba115309b93b4b55b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c98f008c6d416ba115309b93b4b55b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } }
I20260620 09:20:16.296079 18991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:16.296124 18998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:16.296628 18998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:16.296689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 18919
I20260620 09:20:16.296797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:16.296984 392 external_mini_cluster.cc:1428] Setting key 490e347eb312023f9d02b24e136eeba9
I20260620 09:20:16.298589 18998 catalog_manager.cc:1347] Generated new cluster ID: 1d2b23bf0fba40e7bba22a65f563f576
I20260620 09:20:16.298660 18998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:16.302762 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.297423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.0.98.62:41415)
0620 09:20:16.297594 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.297602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.297659 (+ 57us) server_negotiation.cc:407] Connection header received
0620 09:20:16.297767 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.297774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.297841 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.297961 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:16.298300 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.299654 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.300679 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.301305 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.301420 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.302076 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:16.302081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:16.302081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:16.302299 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:16.302346 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:16.302362 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:16.302428 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.302485 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.302561 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260620 09:20:16.305897 18998 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:16.306368 18998 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:16.314229 18998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b: Generated new TSK 0
I20260620 09:20:16.314445 18998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:16.320950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41415
--builtin_ntp_servers=127.0.98.20:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:20:16.400077 19010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:16.400296 19010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:16.400318 19010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:16.401489 19010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:16.401530 19010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:16.401556 19010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:20:16.402926 19010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:16.403155 19010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:16.403445 19010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:16.408143 19016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:16.408157 19015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:16.408326 19010 server_base.cc:1034] running on GCE node
W20260620 09:20:16.408428 19018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:16.408627 19010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:16.408903 19010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:16.410077 19010 hybrid_clock.cc:648] HybridClock initialized: now 1781947216410053 us; error 37 us; skew 500 ppm
I20260620 09:20:16.411491 19010 webserver.cc:460] Webserver started at http://127.0.98.1:42319/ using document root <none> and password file <none>
I20260620 09:20:16.411772 19010 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:16.411828 19010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:16.411958 19010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:16.413261 19010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "7b372bae3ca747b597eabb2b4350f5f5"
format_stamp: "Formatted at 2026-06-20 09:20:16 on dist-test-slave-rq8c"
server_key: "a7b786efe7be584a6272ee62575be186"
server_key_iv: "7460df800dcb6ea96e89c37a311acbd0"
server_key_version: "encryptionkey@0"
I20260620 09:20:16.413659 19010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7b372bae3ca747b597eabb2b4350f5f5"
format_stamp: "Formatted at 2026-06-20 09:20:16 on dist-test-slave-rq8c"
server_key: "a7b786efe7be584a6272ee62575be186"
server_key_iv: "7460df800dcb6ea96e89c37a311acbd0"
server_key_version: "encryptionkey@0"
I20260620 09:20:16.415129 19010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:16.415943 19024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.416096 19010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.416190 19010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "7b372bae3ca747b597eabb2b4350f5f5"
format_stamp: "Formatted at 2026-06-20 09:20:16 on dist-test-slave-rq8c"
server_key: "a7b786efe7be584a6272ee62575be186"
server_key_iv: "7460df800dcb6ea96e89c37a311acbd0"
server_key_version: "encryptionkey@0"
I20260620 09:20:16.416261 19010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:16.423830 19010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:16.424077 19010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:16.424170 19010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:16.424328 19010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:16.424573 19010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:16.424602 19010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.424626 19010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:16.424643 19010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.429996 19010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39861
I20260620 09:20:16.430047 19137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39861 every 8 connection(s)
I20260620 09:20:16.430339 19010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:20:16.435326 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.430995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46523 (local address 127.0.98.62:41415)
0620 09:20:16.431043 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.431045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.431253 (+ 208us) server_negotiation.cc:407] Connection header received
0620 09:20:16.431817 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.431818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.431833 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.431884 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:16.432564 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.433071 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.434034 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.434229 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.434284 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.435071 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:16.435073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:16.435073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:16.435078 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:16.435094 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:16.435099 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:16.435143 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.435214 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.435244 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 09:20:16.435588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 19010
I20260620 09:20:16.435711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:20:16.435889 392 external_mini_cluster.cc:1428] Setting key 8d9dacc5cd9472604858c4487d71cbac
I20260620 09:20:16.436116 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.430970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.98.1:46523)
0620 09:20:16.431146 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:16.431177 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:16.431724 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:16.431950 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:16.431954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:16.431963 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:16.432470 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.432474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.433152 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.433153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.433932 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.433935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.434302 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.434303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.434439 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.434945 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:20:16.434945 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:16.434972 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:20:16.434995 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:16.435149 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:16.435154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:16.435157 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:16.435218 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 09:20:16.435327 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260620 09:20:16.436671 19138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41415
I20260620 09:20:16.436769 19138 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:16.436813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41415
--builtin_ntp_servers=127.0.98.20:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:20:16.436971 19138 heartbeater.cc:511] Master 127.0.98.62:41415 requested a full tablet report, sending...
I20260620 09:20:16.437469 18949 ts_manager.cc:194] Registered new tserver with Master: 7b372bae3ca747b597eabb2b4350f5f5 (127.0.98.1:39861)
I20260620 09:20:16.438053 18949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46523
W20260620 09:20:16.513652 19141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:16.513851 19141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:16.513871 19141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:16.515005 19141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:16.515043 19141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:16.515067 19141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:20:16.516409 19141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:16.516611 19141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:16.516860 19141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:16.521414 19146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:16.521533 19147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:16.521652 19141 server_base.cc:1034] running on GCE node
W20260620 09:20:16.521422 19149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:16.521883 19141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:16.522123 19141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:16.523249 19141 hybrid_clock.cc:648] HybridClock initialized: now 1781947216523237 us; error 30 us; skew 500 ppm
I20260620 09:20:16.524417 19141 webserver.cc:460] Webserver started at http://127.0.98.2:37647/ using document root <none> and password file <none>
I20260620 09:20:16.524628 19141 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:16.524672 19141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:16.524797 19141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:16.525806 19141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "89bddde899a64237967fce534e5d8522"
format_stamp: "Formatted at 2026-06-20 09:20:16 on dist-test-slave-rq8c"
server_key: "876d96ea730a855634b7620f15d04472"
server_key_iv: "8fd6f8cb457a71b021059fe597b5b483"
server_key_version: "encryptionkey@0"
I20260620 09:20:16.526129 19141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "89bddde899a64237967fce534e5d8522"
format_stamp: "Formatted at 2026-06-20 09:20:16 on dist-test-slave-rq8c"
server_key: "876d96ea730a855634b7620f15d04472"
server_key_iv: "8fd6f8cb457a71b021059fe597b5b483"
server_key_version: "encryptionkey@0"
I20260620 09:20:16.527254 19141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:16.527896 19155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.528028 19141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.528100 19141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "89bddde899a64237967fce534e5d8522"
format_stamp: "Formatted at 2026-06-20 09:20:16 on dist-test-slave-rq8c"
server_key: "876d96ea730a855634b7620f15d04472"
server_key_iv: "8fd6f8cb457a71b021059fe597b5b483"
server_key_version: "encryptionkey@0"
I20260620 09:20:16.528162 19141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:16.555334 19141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:16.555644 19141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:16.555771 19141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:16.556002 19141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:16.556317 19141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:16.556349 19141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.556371 19141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:16.556393 19141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.561888 19141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43819
I20260620 09:20:16.561957 19268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43819 every 8 connection(s)
I20260620 09:20:16.562250 19141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:20:16.566788 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.562919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59047 (local address 127.0.98.62:41415)
0620 09:20:16.562973 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.562974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.563159 (+ 185us) server_negotiation.cc:407] Connection header received
0620 09:20:16.563651 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.563652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.563662 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.563722 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:16.564316 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.564793 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.565663 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.565846 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.565887 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.566583 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:16.566584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:16.566585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:16.566588 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:16.566603 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:16.566607 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:16.566643 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.566705 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.566730 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:20:16.566984 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.562878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.98.2:59047)
0620 09:20:16.563049 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:16.563088 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:16.563545 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:16.563776 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:16.563779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:16.563787 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:16.564254 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.564258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.564859 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.564860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.565608 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.565610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.565903 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.565903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.566002 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.566456 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:20:16.566456 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:16.566484 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:20:16.566505 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:16.566654 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:16.566656 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:16.566658 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:20:16.566702 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 09:20:16.566777 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260620 09:20:16.567417 19269 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41415
I20260620 09:20:16.567492 19269 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:16.567662 19269 heartbeater.cc:511] Master 127.0.98.62:41415 requested a full tablet report, sending...
I20260620 09:20:16.567981 18949 ts_manager.cc:194] Registered new tserver with Master: 89bddde899a64237967fce534e5d8522 (127.0.98.2:43819)
I20260620 09:20:16.568325 18949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:59047
I20260620 09:20:16.571187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 19141
I20260620 09:20:16.571273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:20:16.571455 392 external_mini_cluster.cc:1428] Setting key ad47bcc05920af7c1e9d48253ffa6e58
I20260620 09:20:16.572389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41415
--builtin_ntp_servers=127.0.98.20:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:20:16.650696 19272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:16.650890 19272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:16.650910 19272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:16.652084 19272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:16.652124 19272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:16.652148 19272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:20:16.653470 19272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:16.653684 19272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:16.653962 19272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:16.658605 19277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:16.658613 19280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:16.658613 19278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:16.658954 19272 server_base.cc:1034] running on GCE node
I20260620 09:20:16.659142 19272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:16.659366 19272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:16.660527 19272 hybrid_clock.cc:648] HybridClock initialized: now 1781947216660510 us; error 27 us; skew 500 ppm
I20260620 09:20:16.665693 19272 webserver.cc:460] Webserver started at http://127.0.98.3:35287/ using document root <none> and password file <none>
I20260620 09:20:16.666002 19272 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:16.666074 19272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:16.666225 19272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:16.667346 19272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "095b6596857840aa9ac90af201cce121"
format_stamp: "Formatted at 2026-06-20 09:20:16 on dist-test-slave-rq8c"
server_key: "a1d94facdce8ba045de3fcf5b1261417"
server_key_iv: "c14ce1015639f851fd2ca3b9a1abc415"
server_key_version: "encryptionkey@0"
I20260620 09:20:16.667750 19272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "095b6596857840aa9ac90af201cce121"
format_stamp: "Formatted at 2026-06-20 09:20:16 on dist-test-slave-rq8c"
server_key: "a1d94facdce8ba045de3fcf5b1261417"
server_key_iv: "c14ce1015639f851fd2ca3b9a1abc415"
server_key_version: "encryptionkey@0"
I20260620 09:20:16.669106 19272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:20:16.669798 19286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.669956 19272 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:20:16.670037 19272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "095b6596857840aa9ac90af201cce121"
format_stamp: "Formatted at 2026-06-20 09:20:16 on dist-test-slave-rq8c"
server_key: "a1d94facdce8ba045de3fcf5b1261417"
server_key_iv: "c14ce1015639f851fd2ca3b9a1abc415"
server_key_version: "encryptionkey@0"
I20260620 09:20:16.670104 19272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:16.693161 19272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:16.693477 19272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:16.693601 19272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:16.693812 19272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:16.694115 19272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:16.694146 19272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.694175 19272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:16.694197 19272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:16.699806 19272 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41321
I20260620 09:20:16.699860 19399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41321 every 8 connection(s)
I20260620 09:20:16.700160 19272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:20:16.705070 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.700823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33933 (local address 127.0.98.62:41415)
0620 09:20:16.700879 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.700881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.701104 (+ 223us) server_negotiation.cc:407] Connection header received
0620 09:20:16.701533 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.701534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.701544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.701628 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:16.702293 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.702800 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.703790 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.703978 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.704031 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.704846 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:16.704848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:16.704848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:16.704853 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:16.704869 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:16.704873 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:16.704925 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.704982 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.705008 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 09:20:16.705282 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.700796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.98.3:33933)
0620 09:20:16.700965 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:16.701004 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:16.701457 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:16.701678 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:16.701681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:16.701688 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:16.702186 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.702190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.702874 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.702875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.703718 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.703721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.704040 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.704041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.704146 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.704685 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:20:16.704686 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:16.704714 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:20:16.704737 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:16.704931 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:16.704934 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:16.704937 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:16.704983 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 09:20:16.705067 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260620 09:20:16.705721 19400 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41415
I20260620 09:20:16.705816 19400 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:16.705993 19400 heartbeater.cc:511] Master 127.0.98.62:41415 requested a full tablet report, sending...
I20260620 09:20:16.706370 18949 ts_manager.cc:194] Registered new tserver with Master: 095b6596857840aa9ac90af201cce121 (127.0.98.3:41321)
I20260620 09:20:16.706710 18949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33933
I20260620 09:20:16.706993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 19272
I20260620 09:20:16.707075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:20:16.707252 392 external_mini_cluster.cc:1428] Setting key 8bf36586f6c2902e77c9d6df9b0c3e3d
I20260620 09:20:16.708072 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:20:16.711732 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.709296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.0.98.62:41415)
0620 09:20:16.709342 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.709344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.709470 (+ 126us) server_negotiation.cc:407] Connection header received
0620 09:20:16.709505 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.709505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.709517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.709552 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:16.709681 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.710139 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.710791 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.710957 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.710999 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.711499 (+ 500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:16.711500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:16.711500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:16.711504 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:16.711515 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:16.711519 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:16.711581 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.711642 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.711653 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 09:20:16.713043 18949 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:20:16.714519 18949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:20:16.720575 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.716418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41321 (local address 127.0.0.1:40942)
0620 09:20:16.716881 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:16.716889 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:16.716927 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:16.717112 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:16.717116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:16.717118 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:16.717191 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.717193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.718504 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.718505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.719325 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.719327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.720240 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.720241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.720361 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.720378 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 09:20:16.720444 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:20:16.720496 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":5,"spinlock_wait_cycles":896,"thread_start_us":78,"threads_started":1}
I20260620 09:20:16.720584 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.716486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.0.98.3:41321)
0620 09:20:16.716909 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.716914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.716925 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:20:16.717025 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.717028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.717041 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.717106 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:16.717335 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.718428 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.719403 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.720156 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.720230 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.720373 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.720456 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.720494 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":99,"threads_started":1}
I20260620 09:20:16.720737 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.716559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.0.98.2:43819)
0620 09:20:16.716703 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.716708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.716940 (+ 232us) server_negotiation.cc:407] Connection header received
0620 09:20:16.716981 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.716983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.716999 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.717067 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:16.717260 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.718387 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.719664 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.720215 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.720317 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.720412 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.720615 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.720655 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260620 09:20:16.720867 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.716604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39861 (local address 127.0.0.1:38392)
0620 09:20:16.716805 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:16.716815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:16.717000 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:16.717328 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:16.717329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:16.717331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:16.717369 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.717370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.718261 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.718262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.719473 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.719476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.720433 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.720434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.720494 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.720498 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:20:16.720546 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 09:20:16.720587 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":28,"thread_start_us":93,"threads_started":1}
I20260620 09:20:16.720963 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.716417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43819 (local address 127.0.0.1:49600)
0620 09:20:16.716740 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:16.716778 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:16.716860 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:16.717136 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:16.717136 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:16.717139 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:16.717189 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.717191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.718469 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.718470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.719564 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.719566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.720444 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.720445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.720499 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.720502 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:16.720545 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 09:20:16.720578 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":19,"thread_start_us":60,"threads_started":1}
I20260620 09:20:16.721374 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.716705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.0.98.1:39861)
0620 09:20:16.716877 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.716882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.716892 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:20:16.717094 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.717096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.717112 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.717261 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:16.717440 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.718163 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.719551 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.720258 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.720315 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.720401 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.720627 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.721302 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260620 09:20:16.721390 19334 tablet_service.cc:1467] Processing CreateTablet for tablet b379e74fa09c46248f7ae8b5c929993a (DEFAULT_TABLE table=test-table [id=8ef6a059422942a28a20bea4e4e650f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:16.721750 19334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b379e74fa09c46248f7ae8b5c929993a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:16.721752 19203 tablet_service.cc:1467] Processing CreateTablet for tablet b379e74fa09c46248f7ae8b5c929993a (DEFAULT_TABLE table=test-table [id=8ef6a059422942a28a20bea4e4e650f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:16.722069 19203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b379e74fa09c46248f7ae8b5c929993a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:16.722100 19072 tablet_service.cc:1467] Processing CreateTablet for tablet b379e74fa09c46248f7ae8b5c929993a (DEFAULT_TABLE table=test-table [id=8ef6a059422942a28a20bea4e4e650f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:16.722391 19072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b379e74fa09c46248f7ae8b5c929993a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:16.725147 19414 tablet_bootstrap.cc:492] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5: Bootstrap starting.
I20260620 09:20:16.725203 19416 tablet_bootstrap.cc:492] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522: Bootstrap starting.
I20260620 09:20:16.725701 19414 tablet_bootstrap.cc:654] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:16.725709 19416 tablet_bootstrap.cc:654] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:16.725927 19414 log.cc:824] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:16.725976 19416 log.cc:824] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:16.726295 19415 tablet_bootstrap.cc:492] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121: Bootstrap starting.
I20260620 09:20:16.726645 19416 tablet_bootstrap.cc:492] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522: No bootstrap required, opened a new log
I20260620 09:20:16.726658 19414 tablet_bootstrap.cc:492] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5: No bootstrap required, opened a new log
I20260620 09:20:16.726716 19414 ts_tablet_manager.cc:1397] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:20:16.726716 19416 ts_tablet_manager.cc:1397] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:20:16.726810 19415 tablet_bootstrap.cc:654] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:16.727053 19415 log.cc:824] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:16.727643 19415 tablet_bootstrap.cc:492] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121: No bootstrap required, opened a new log
I20260620 09:20:16.727705 19415 ts_tablet_manager.cc:1397] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:20:16.729161 19416 raft_consensus.cc:348] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bddde899a64237967fce534e5d8522" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } } peers { permanent_uuid: "095b6596857840aa9ac90af201cce121" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "7b372bae3ca747b597eabb2b4350f5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39861 } }
I20260620 09:20:16.729161 19414 raft_consensus.cc:348] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bddde899a64237967fce534e5d8522" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } } peers { permanent_uuid: "095b6596857840aa9ac90af201cce121" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "7b372bae3ca747b597eabb2b4350f5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39861 } }
I20260620 09:20:16.729328 19414 raft_consensus.cc:374] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:16.729328 19416 raft_consensus.cc:374] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:16.729363 19414 raft_consensus.cc:729] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b372bae3ca747b597eabb2b4350f5f5, State: Initialized, Role: FOLLOWER
I20260620 09:20:16.729401 19416 raft_consensus.cc:729] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89bddde899a64237967fce534e5d8522, State: Initialized, Role: FOLLOWER
I20260620 09:20:16.729527 19416 consensus_queue.cc:260] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bddde899a64237967fce534e5d8522" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } } peers { permanent_uuid: "095b6596857840aa9ac90af201cce121" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "7b372bae3ca747b597eabb2b4350f5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39861 } }
I20260620 09:20:16.729517 19414 consensus_queue.cc:260] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bddde899a64237967fce534e5d8522" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } } peers { permanent_uuid: "095b6596857840aa9ac90af201cce121" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "7b372bae3ca747b597eabb2b4350f5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39861 } }
I20260620 09:20:16.729552 19415 raft_consensus.cc:348] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bddde899a64237967fce534e5d8522" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } } peers { permanent_uuid: "095b6596857840aa9ac90af201cce121" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "7b372bae3ca747b597eabb2b4350f5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39861 } }
I20260620 09:20:16.729714 19415 raft_consensus.cc:374] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:16.729764 19415 raft_consensus.cc:729] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095b6596857840aa9ac90af201cce121, State: Initialized, Role: FOLLOWER
I20260620 09:20:16.729775 19416 ts_tablet_manager.cc:1428] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:20:16.729775 19414 ts_tablet_manager.cc:1428] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:20:16.729871 19415 consensus_queue.cc:260] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bddde899a64237967fce534e5d8522" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } } peers { permanent_uuid: "095b6596857840aa9ac90af201cce121" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "7b372bae3ca747b597eabb2b4350f5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39861 } }
I20260620 09:20:16.729928 19269 heartbeater.cc:503] Master 127.0.98.62:41415 was elected leader, sending a full tablet report...
I20260620 09:20:16.730111 19415 ts_tablet_manager.cc:1428] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:20:16.730149 19400 heartbeater.cc:503] Master 127.0.98.62:41415 was elected leader, sending a full tablet report...
I20260620 09:20:16.730187 19138 heartbeater.cc:503] Master 127.0.98.62:41415 was elected leader, sending a full tablet report...
I20260620 09:20:16.760314 19422 raft_consensus.cc:482] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:20:16.760452 19422 raft_consensus.cc:504] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bddde899a64237967fce534e5d8522" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } } peers { permanent_uuid: "095b6596857840aa9ac90af201cce121" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "7b372bae3ca747b597eabb2b4350f5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39861 } }
I20260620 09:20:16.760795 19422 leader_election.cc:288] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89bddde899a64237967fce534e5d8522 (127.0.98.2:43819), 7b372bae3ca747b597eabb2b4350f5f5 (127.0.98.1:39861)
I20260620 09:20:16.764580 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.761172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41671 (local address 127.0.98.2:43819)
0620 09:20:16.761215 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.761217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.761225 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:16.761254 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.761255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.761266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.761318 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:16.761452 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.762174 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.763587 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.764182 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.764257 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.764267 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.764466 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.764505 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:20:16.764580 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.761216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33439 (local address 127.0.98.1:39861)
0620 09:20:16.761259 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.761262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.761384 (+ 122us) server_negotiation.cc:407] Connection header received
0620 09:20:16.761441 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.761442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.761454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.761485 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:16.761669 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.762338 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.763585 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.764182 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.764257 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.764267 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.764466 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.764505 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:20:16.764629 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.761022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43819 (local address 127.0.98.3:41671)
0620 09:20:16.761093 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:16.761106 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:16.761173 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:16.761325 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:16.761326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:16.761327 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:16.761394 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.761396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.762263 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.762264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.763497 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.763499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.764294 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.764295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.764382 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.764387 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:20:16.764462 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:20:16.764535 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":2}
I20260620 09:20:16.764643 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.761127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39861 (local address 127.0.98.3:33439)
0620 09:20:16.761295 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:16.761319 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:16.761377 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:16.761524 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:16.761525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:16.761526 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:16.761594 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.761596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.762408 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.762409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.763505 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:16.763507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.764295 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.764296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:16.764390 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.764393 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:16.764462 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 09:20:16.764528 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":27,"thread_start_us":122,"threads_started":1}
I20260620 09:20:16.765012 19092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b379e74fa09c46248f7ae8b5c929993a" candidate_uuid: "095b6596857840aa9ac90af201cce121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b372bae3ca747b597eabb2b4350f5f5" is_pre_election: true
I20260620 09:20:16.765030 19222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b379e74fa09c46248f7ae8b5c929993a" candidate_uuid: "095b6596857840aa9ac90af201cce121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89bddde899a64237967fce534e5d8522" is_pre_election: true
I20260620 09:20:16.765151 19222 raft_consensus.cc:2454] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 095b6596857840aa9ac90af201cce121 in term 0.
I20260620 09:20:16.765162 19092 raft_consensus.cc:2454] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 095b6596857840aa9ac90af201cce121 in term 0.
I20260620 09:20:16.765388 19289 leader_election.cc:302] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 095b6596857840aa9ac90af201cce121, 89bddde899a64237967fce534e5d8522; no voters:
I20260620 09:20:16.765504 19422 raft_consensus.cc:2781] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:20:16.765558 19422 raft_consensus.cc:482] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:20:16.765580 19422 raft_consensus.cc:3037] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:16.766239 19422 raft_consensus.cc:504] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bddde899a64237967fce534e5d8522" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } } peers { permanent_uuid: "095b6596857840aa9ac90af201cce121" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "7b372bae3ca747b597eabb2b4350f5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39861 } }
I20260620 09:20:16.766346 19422 leader_election.cc:288] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [CANDIDATE]: Term 1 election: Requested vote from peers 89bddde899a64237967fce534e5d8522 (127.0.98.2:43819), 7b372bae3ca747b597eabb2b4350f5f5 (127.0.98.1:39861)
I20260620 09:20:16.766516 19222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b379e74fa09c46248f7ae8b5c929993a" candidate_uuid: "095b6596857840aa9ac90af201cce121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89bddde899a64237967fce534e5d8522"
I20260620 09:20:16.766561 19092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b379e74fa09c46248f7ae8b5c929993a" candidate_uuid: "095b6596857840aa9ac90af201cce121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b372bae3ca747b597eabb2b4350f5f5"
I20260620 09:20:16.766600 19222 raft_consensus.cc:3037] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:16.766626 19092 raft_consensus.cc:3037] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:16.767441 19092 raft_consensus.cc:2454] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 095b6596857840aa9ac90af201cce121 in term 1.
I20260620 09:20:16.767441 19222 raft_consensus.cc:2454] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 095b6596857840aa9ac90af201cce121 in term 1.
I20260620 09:20:16.767642 19289 leader_election.cc:302] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 095b6596857840aa9ac90af201cce121, 7b372bae3ca747b597eabb2b4350f5f5; no voters:
I20260620 09:20:16.767721 19422 raft_consensus.cc:2781] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:16.767886 19422 raft_consensus.cc:686] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 1 LEADER]: Becoming Leader. State: Replica: 095b6596857840aa9ac90af201cce121, State: Running, Role: LEADER
I20260620 09:20:16.767966 19422 consensus_queue.cc:237] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bddde899a64237967fce534e5d8522" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } } peers { permanent_uuid: "095b6596857840aa9ac90af201cce121" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } } peers { permanent_uuid: "7b372bae3ca747b597eabb2b4350f5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39861 } }
I20260620 09:20:16.768754 18950 catalog_manager.cc:5495] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 reported cstate change: term changed from 0 to 1, leader changed from <none> to 095b6596857840aa9ac90af201cce121 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "095b6596857840aa9ac90af201cce121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89bddde899a64237967fce534e5d8522" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "095b6596857840aa9ac90af201cce121" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b372bae3ca747b597eabb2b4350f5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39861 } health_report { overall_health: UNKNOWN } } }
I20260620 09:20:16.777652 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.775439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.0.98.3:41321)
0620 09:20:16.775485 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:16.775486 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:16.775494 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:16.775576 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:16.775577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:16.775592 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:16.775681 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:16.775830 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.776276 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.777030 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:16.777217 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:16.777270 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:16.777380 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:16.777462 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:16.777526 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:16.777571 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 09:20:16.777578 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:20:16.779297 19092 raft_consensus.cc:1261] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5 [term 1 FOLLOWER]: Refusing update from remote peer 095b6596857840aa9ac90af201cce121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:16.779297 19222 raft_consensus.cc:1261] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522 [term 1 FOLLOWER]: Refusing update from remote peer 095b6596857840aa9ac90af201cce121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:16.779641 19422 consensus_queue.cc:1035] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b372bae3ca747b597eabb2b4350f5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:16.779788 19424 consensus_queue.cc:1035] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89bddde899a64237967fce534e5d8522" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:16.782388 19429 mvcc.cc:205] Tried to move back new op lower bound from 7298855799925596160 to 7298855799882293248. Current Snapshot: MvccSnapshot[applied={T|T < 7298855799925596160}]
I20260620 09:20:16.782666 19428 mvcc.cc:205] Tried to move back new op lower bound from 7298855799925596160 to 7298855799882293248. Current Snapshot: MvccSnapshot[applied={T|T < 7298855799925596160}]
I20260620 09:20:16.784243 18950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:20:16.784335 18950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:20:16.785010 18950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61c98f008c6d416ba115309b93b4b55b: Sending DeleteTablet for 3 replicas of tablet b379e74fa09c46248f7ae8b5c929993a
I20260620 09:20:16.785351 19072 tablet_service.cc:1514] Processing DeleteTablet for tablet b379e74fa09c46248f7ae8b5c929993a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:16 UTC) from {username='slave'} at 127.0.0.1:38392
I20260620 09:20:16.785357 19334 tablet_service.cc:1514] Processing DeleteTablet for tablet b379e74fa09c46248f7ae8b5c929993a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:16 UTC) from {username='slave'} at 127.0.0.1:40942
I20260620 09:20:16.785382 19203 tablet_service.cc:1514] Processing DeleteTablet for tablet b379e74fa09c46248f7ae8b5c929993a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:16 UTC) from {username='slave'} at 127.0.0.1:49600
I20260620 09:20:16.785569 19437 tablet_replica.cc:330] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522: stopping tablet replica
I20260620 09:20:16.785656 19437 raft_consensus.cc:2229] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:20:16.785758 19437 raft_consensus.cc:2258] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:16.785785 19439 tablet_replica.cc:330] T b379e74fa09c46248f7ae8b5c929993a P 7b372bae3ca747b597eabb2b4350f5f5: stopping tablet replica
I20260620 09:20:16.785889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 19010
I20260620 09:20:16.786331 19437 ts_tablet_manager.cc:1905] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:16.787556 19438 tablet_replica.cc:330] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121: stopping tablet replica
I20260620 09:20:16.787710 19437 ts_tablet_manager.cc:1918] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:20:16.787774 19437 log.cc:1196] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/b379e74fa09c46248f7ae8b5c929993a
I20260620 09:20:16.787976 19437 ts_tablet_manager.cc:1939] T b379e74fa09c46248f7ae8b5c929993a P 89bddde899a64237967fce534e5d8522: Deleting consensus metadata
I20260620 09:20:16.788286 19438 raft_consensus.cc:2229] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:20:16.788323 18936 catalog_manager.cc:4841] TS 89bddde899a64237967fce534e5d8522 (127.0.98.2:43819): tablet b379e74fa09c46248f7ae8b5c929993a (table test-table [id=8ef6a059422942a28a20bea4e4e650f7]) successfully deleted
I20260620 09:20:16.788414 19438 raft_consensus.cc:2258] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:16.788748 19438 ts_tablet_manager.cc:1905] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:16.789944 19438 ts_tablet_manager.cc:1918] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:20:16.790002 19438 log.cc:1196] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/b379e74fa09c46248f7ae8b5c929993a
I20260620 09:20:16.790184 19438 ts_tablet_manager.cc:1939] T b379e74fa09c46248f7ae8b5c929993a P 095b6596857840aa9ac90af201cce121: Deleting consensus metadata
I20260620 09:20:16.790504 18937 catalog_manager.cc:4841] TS 095b6596857840aa9ac90af201cce121 (127.0.98.3:41321): tablet b379e74fa09c46248f7ae8b5c929993a (table test-table [id=8ef6a059422942a28a20bea4e4e650f7]) successfully deleted
W20260620 09:20:16.793202 18936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39861 (error 108)
I20260620 09:20:16.793411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 19141
I20260620 09:20:16.794080 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:16.793536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39861 (local address 127.0.0.1:38402)
0620 09:20:16.794006 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:16.794046 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451}
W20260620 09:20:16.794206 18936 catalog_manager.cc:4584] TS 7b372bae3ca747b597eabb2b4350f5f5 (127.0.98.1:39861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b379e74fa09c46248f7ae8b5c929993a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39861: connect: Connection refused (error 111)
I20260620 09:20:16.798324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 19272
I20260620 09:20:16.803866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 18919
2026-06-20T09:20:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (677 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:16 dist-test-slave-rq8c krb5kdc[19443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:16 dist-test-slave-rq8c krb5kdc[19443](info): set up 2 sockets
Jun 20 09:20:16 dist-test-slave-rq8c krb5kdc[19443](info): commencing operation
krb5kdc: starting...
W20260620 09:20:17.747153 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:17 dist-test-slave-rq8c krb5kdc[19443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:20:17.852008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43185
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:17.930586 19456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:17.930805 19456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:17.930827 19456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:17.931964 19456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:17.931996 19456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:17.932009 19456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:17.932027 19456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:17.932041 19456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:17.933447 19456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:17.933646 19456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:17.933815 19456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:17.938493 19464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:17.938484 19461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:17.938481 19462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:17.938686 19456 server_base.cc:1034] running on GCE node
I20260620 09:20:17.938988 19456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:17.939266 19456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:17.940402 19456 hybrid_clock.cc:648] HybridClock initialized: now 1781947217940386 us; error 27 us; skew 500 ppm
Jun 20 09:20:17 dist-test-slave-rq8c krb5kdc[19443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:17.942749 19456 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:20:17.943143 19456 webserver.cc:460] Webserver started at http://127.0.98.62:33749/ using document root <none> and password file <none>
I20260620 09:20:17.943351 19456 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:17.943396 19456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:17.943516 19456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:17.944576 19456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "d1d58dd466fa490796b11534505135d6"
format_stamp: "Formatted at 2026-06-20 09:20:17 on dist-test-slave-rq8c"
server_key: "926886663c71fd2ff7e5cd80d0494596"
server_key_iv: "27b527abeca91bdb0bc776200b8a9129"
server_key_version: "encryptionkey@0"
I20260620 09:20:17.944906 19456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "d1d58dd466fa490796b11534505135d6"
format_stamp: "Formatted at 2026-06-20 09:20:17 on dist-test-slave-rq8c"
server_key: "926886663c71fd2ff7e5cd80d0494596"
server_key_iv: "27b527abeca91bdb0bc776200b8a9129"
server_key_version: "encryptionkey@0"
I20260620 09:20:17.946105 19456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:20:17.946784 19471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:17.946935 19456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:20:17.947013 19456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "d1d58dd466fa490796b11534505135d6"
format_stamp: "Formatted at 2026-06-20 09:20:17 on dist-test-slave-rq8c"
server_key: "926886663c71fd2ff7e5cd80d0494596"
server_key_iv: "27b527abeca91bdb0bc776200b8a9129"
server_key_version: "encryptionkey@0"
I20260620 09:20:17.947078 19456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:17.966243 19456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:17.968364 19456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:17.968502 19456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:17.972124 19456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43185
I20260620 09:20:17.972169 19523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43185 every 8 connection(s)
I20260620 09:20:17.972484 19456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:17.973395 19524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:17.975764 19524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6: Bootstrap starting.
I20260620 09:20:17.976325 19524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:17.976533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 19456
I20260620 09:20:17.976629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:17.976619 19524 log.cc:824] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:17.976823 392 external_mini_cluster.cc:1428] Setting key b842ac4c165bd705ddcfe7aafa636fbc
I20260620 09:20:17.977311 19524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6: No bootstrap required, opened a new log
I20260620 09:20:17.978919 19524 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d58dd466fa490796b11534505135d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43185 } }
I20260620 09:20:17.979049 19524 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:17.979071 19524 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d58dd466fa490796b11534505135d6, State: Initialized, Role: FOLLOWER
I20260620 09:20:17.979203 19524 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d58dd466fa490796b11534505135d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43185 } }
I20260620 09:20:17.979295 19524 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:17.979324 19524 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:17.979370 19524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:17.980284 19524 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d58dd466fa490796b11534505135d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43185 } }
I20260620 09:20:17.980371 19524 leader_election.cc:302] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1d58dd466fa490796b11534505135d6; no voters:
I20260620 09:20:17.980506 19524 leader_election.cc:288] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:17.980566 19529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:17.980757 19529 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [term 1 LEADER]: Becoming Leader. State: Replica: d1d58dd466fa490796b11534505135d6, State: Running, Role: LEADER
I20260620 09:20:17.980762 19524 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:17.980862 19529 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d58dd466fa490796b11534505135d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43185 } }
I20260620 09:20:17.981357 19530 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1d58dd466fa490796b11534505135d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d58dd466fa490796b11534505135d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43185 } } }
I20260620 09:20:17.981516 19530 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:17.982014 19538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:17.982365 19531 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1d58dd466fa490796b11534505135d6. Latest consensus state: current_term: 1 leader_uuid: "d1d58dd466fa490796b11534505135d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d58dd466fa490796b11534505135d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43185 } } }
I20260620 09:20:17.982466 19531 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:17.982514 19538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:17.984211 19538 catalog_manager.cc:1347] Generated new cluster ID: 53a61ffc56064bfca3dbbe43f9aa29e3
I20260620 09:20:17.984267 19538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:20:17 dist-test-slave-rq8c krb5kdc[19443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:17.989164 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:17.977330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.0.98.62:43185)
0620 09:20:17.977492 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:17.977501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:17.977533 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:20:17.978927 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:17.978935 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:17.978985 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:17.979135 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:17.979889 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:17.981084 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:17.982132 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:17.982701 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:17.982804 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:17.985574 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:17.985578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:17.985579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:17.985590 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:17.987498 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:17.987898 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:17.987901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:17.987902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:17.988035 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:17.988224 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:17.988225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:17.988225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:17.988574 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:17.988724 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:17.988834 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 09:20:17.988973 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260620 09:20:18.006409 19538 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:18.006969 19538 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:18.013760 19538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6: Generated new TSK 0
I20260620 09:20:18.013945 19538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:20:18.042464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43185
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:18.130270 19550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:18.130476 19550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:18.130507 19550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:18.131692 19550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:18.131733 19550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:18.131767 19550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:20:18.133210 19550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:18.133431 19550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:18.133635 19550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:18.138355 19556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:18.138352 19555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:18.138353 19558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:18.138536 19550 server_base.cc:1034] running on GCE node
I20260620 09:20:18.138751 19550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:18.138960 19550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:18.140092 19550 hybrid_clock.cc:648] HybridClock initialized: now 1781947218140075 us; error 33 us; skew 500 ppm
Jun 20 09:20:18 dist-test-slave-rq8c krb5kdc[19443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:18.142436 19550 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:20:18.142827 19550 webserver.cc:460] Webserver started at http://127.0.98.1:40933/ using document root <none> and password file <none>
I20260620 09:20:18.143013 19550 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:18.143057 19550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:18.143173 19550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:18.144217 19550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "5cfbbda84f1f48a4b664c29f6003ef6d"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "c4d861ba2c047610b4b4bb0f116ada12"
server_key_iv: "c5673812c11cf20175e92931c06baa42"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.144544 19550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5cfbbda84f1f48a4b664c29f6003ef6d"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "c4d861ba2c047610b4b4bb0f116ada12"
server_key_iv: "c5673812c11cf20175e92931c06baa42"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.145717 19550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:20:18.146332 19565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.146487 19550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:18.146564 19550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "5cfbbda84f1f48a4b664c29f6003ef6d"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "c4d861ba2c047610b4b4bb0f116ada12"
server_key_iv: "c5673812c11cf20175e92931c06baa42"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.146620 19550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:18.160924 19550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:18.162657 19550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:18.162783 19550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:18.162972 19550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:18.163254 19550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:18.163285 19550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.163312 19550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:18.163329 19550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.168565 19550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34921
I20260620 09:20:18.168627 19678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34921 every 8 connection(s)
I20260620 09:20:18.168890 19550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:20:18 dist-test-slave-rq8c krb5kdc[19443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:18.176890 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.169499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41331 (local address 127.0.98.62:43185)
0620 09:20:18.169568 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.169571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.169729 (+ 158us) server_negotiation.cc:407] Connection header received
0620 09:20:18.170593 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.170593 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.170607 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.170647 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:18.171185 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.171694 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.172632 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.172838 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.172890 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.174759 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:18.174759 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:18.174760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:18.174765 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:18.175886 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:18.176208 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:18.176208 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:18.176209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:18.176260 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:18.176423 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:18.176424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:18.176424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:18.176514 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:18.176561 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.176756 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.176822 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:20:18.177022 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.169478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43185 (local address 127.0.98.1:41331)
0620 09:20:18.169615 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:18.169657 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:18.170502 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:18.170706 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:18.170709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:18.170945 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:18.171122 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.171126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.171760 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.171761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.172554 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.172556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.172898 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.172899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.172990 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.173431 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:18.173439 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:18.174689 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:18.175949 (+ 1260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:18.175955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:18.175957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:18.176150 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:18.176300 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:18.176301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:18.176301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:18.176384 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:18.176554 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:18.176556 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:18.176667 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 09:20:18.176741 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:20:18.176850 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260620 09:20:18.177368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 19550
I20260620 09:20:18.177467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:20:18.177467 19679 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43185
I20260620 09:20:18.177549 19679 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:18.177664 392 external_mini_cluster.cc:1428] Setting key eef24b90062e5c3a9e9e91253b40f038
I20260620 09:20:18.177723 19679 heartbeater.cc:511] Master 127.0.98.62:43185 requested a full tablet report, sending...
I20260620 09:20:18.178187 19487 ts_manager.cc:194] Registered new tserver with Master: 5cfbbda84f1f48a4b664c29f6003ef6d (127.0.98.1:34921)
I20260620 09:20:18.178766 19487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41331
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:20:18.198275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43185
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:18.275521 19684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:18.275741 19684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:18.275765 19684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:18.276880 19684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:18.276912 19684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:18.276937 19684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:20:18.278352 19684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:18.278528 19684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:18.278705 19684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:20:18.283816 19684 server_base.cc:1034] running on GCE node
W20260620 09:20:18.283798 19689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:18.283775 19690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:18.283792 19692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:18.284160 19684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:18.284404 19684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:18.285542 19684 hybrid_clock.cc:648] HybridClock initialized: now 1781947218285531 us; error 29 us; skew 500 ppm
Jun 20 09:20:18 dist-test-slave-rq8c krb5kdc[19443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:18.287865 19684 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:20:18.288290 19684 webserver.cc:460] Webserver started at http://127.0.98.2:39409/ using document root <none> and password file <none>
I20260620 09:20:18.288498 19684 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:18.288543 19684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:18.288661 19684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:18.289662 19684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "fe940c85c07b455185b6961bbfc7640c"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "4332ca5906d3e51a13df9ae9114fde0d"
server_key_iv: "761e2a77c68ed1979c79f22204e9310e"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.289984 19684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fe940c85c07b455185b6961bbfc7640c"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "4332ca5906d3e51a13df9ae9114fde0d"
server_key_iv: "761e2a77c68ed1979c79f22204e9310e"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.291198 19684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:18.291947 19699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.292135 19684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:20:18.292207 19684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "fe940c85c07b455185b6961bbfc7640c"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "4332ca5906d3e51a13df9ae9114fde0d"
server_key_iv: "761e2a77c68ed1979c79f22204e9310e"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.292280 19684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:18.302285 19684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:18.304250 19684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:18.304376 19684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:18.304572 19684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:18.304837 19684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:18.304868 19684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.304896 19684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:18.304914 19684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.310171 19684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45001
I20260620 09:20:18.310233 19812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45001 every 8 connection(s)
I20260620 09:20:18.310504 19684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:20:18.312441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 19684
I20260620 09:20:18.312548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:20:18.312741 392 external_mini_cluster.cc:1428] Setting key 6918e0732cf9cf3039f5b0c33b65f427
Jun 20 09:20:18 dist-test-slave-rq8c krb5kdc[19443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:18.320150 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.311113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59587 (local address 127.0.98.62:43185)
0620 09:20:18.311166 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.311168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.311358 (+ 190us) server_negotiation.cc:407] Connection header received
0620 09:20:18.312260 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.312262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.312272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.312335 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:18.313120 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.313771 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.315094 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.315384 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.315448 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.317633 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:18.317634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:18.317634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:18.317640 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:18.318848 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:18.319290 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:18.319291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:18.319291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:18.319335 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:18.319554 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:18.319554 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:18.319554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:18.319679 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:18.319724 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.320017 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.320086 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:20:18.320406 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.311082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43185 (local address 127.0.98.2:59587)
0620 09:20:18.311247 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:18.311281 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:18.312176 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:18.312438 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:18.312442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:18.312782 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:18.313034 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.313040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.313849 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.313850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.315023 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.315026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.315455 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.315455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.315543 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.316015 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:18.316023 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:18.317544 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:18.318923 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:18.318929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:18.318931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:18.319225 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:18.319374 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:18.319375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:18.319375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:18.319501 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:18.319732 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:18.319738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:18.319897 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 09:20:18.320009 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 09:20:18.320179 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:20:18.320858 19813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43185
I20260620 09:20:18.320940 19813 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:18.321103 19813 heartbeater.cc:511] Master 127.0.98.62:43185 requested a full tablet report, sending...
I20260620 09:20:18.321462 19487 ts_manager.cc:194] Registered new tserver with Master: fe940c85c07b455185b6961bbfc7640c (127.0.98.2:45001)
I20260620 09:20:18.321898 19487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:20:18.335557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43185
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:18.412528 19818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:18.412714 19818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:18.412734 19818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:18.413839 19818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:18.413875 19818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:18.413903 19818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:20:18.415328 19818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:18.415526 19818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:18.415740 19818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:18.420285 19823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:18.420287 19824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:18.420261 19826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:18.420424 19818 server_base.cc:1034] running on GCE node
I20260620 09:20:18.420632 19818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:18.420843 19818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:18.421959 19818 hybrid_clock.cc:648] HybridClock initialized: now 1781947218421946 us; error 29 us; skew 500 ppm
Jun 20 09:20:18 dist-test-slave-rq8c krb5kdc[19443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:18.424276 19818 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:20:18.424660 19818 webserver.cc:460] Webserver started at http://127.0.98.3:45669/ using document root <none> and password file <none>
I20260620 09:20:18.424850 19818 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:18.424896 19818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:18.425014 19818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:18.425990 19818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "fbab88bf06264427baefc814d0d5e876"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "a7d45f5cb1d50e29d45fb43a281f4eaa"
server_key_iv: "365aadfd60dd88aeafa7a663a2eeed5c"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.426301 19818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fbab88bf06264427baefc814d0d5e876"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "a7d45f5cb1d50e29d45fb43a281f4eaa"
server_key_iv: "365aadfd60dd88aeafa7a663a2eeed5c"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.427472 19818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:18.428208 19833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.428366 19818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.428444 19818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "fbab88bf06264427baefc814d0d5e876"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "a7d45f5cb1d50e29d45fb43a281f4eaa"
server_key_iv: "365aadfd60dd88aeafa7a663a2eeed5c"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.428508 19818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:18.440510 19818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:18.442385 19818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:18.442521 19818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:18.442728 19818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:18.443001 19818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:18.443033 19818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.443061 19818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:18.443081 19818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.448326 19818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43937
I20260620 09:20:18.448374 19946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43937 every 8 connection(s)
I20260620 09:20:18.448714 19818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:20:18.449479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 19818
I20260620 09:20:18.449582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:20:18.449757 392 external_mini_cluster.cc:1428] Setting key 8dfe75769bff2403fe759e1002356480
Jun 20 09:20:18 dist-test-slave-rq8c krb5kdc[19443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:18.457358 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.449646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59265 (local address 127.0.98.62:43185)
0620 09:20:18.449693 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.449695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.449706 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:20:18.450695 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.450696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.450708 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.450774 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:18.451354 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.451904 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.452861 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.453053 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.453107 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.455087 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:18.455088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:18.455089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:18.455095 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:18.456250 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:18.456605 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:18.456606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:18.456606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:18.456647 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:18.456820 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:18.456820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:18.456821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:18.456910 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:18.456954 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.457240 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.457294 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:20:18.457542 19949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.449323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43185 (local address 127.0.98.3:59265)
0620 09:20:18.449487 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:18.449522 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:18.450605 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:18.450842 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:18.450846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:18.451087 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:18.451267 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.451271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.451983 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.451984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.452779 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.452782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.453118 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.453119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.453243 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.453666 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:18.453673 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:18.455010 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:18.456315 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:18.456322 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:18.456323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:18.456543 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:18.456689 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:18.456690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:18.456690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:18.456779 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:18.456990 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:18.456997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:18.457125 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 09:20:18.457221 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 09:20:18.457343 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260620 09:20:18.457952 19947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43185
I20260620 09:20:18.458035 19947 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:18.458184 19947 heartbeater.cc:511] Master 127.0.98.62:43185 requested a full tablet report, sending...
I20260620 09:20:18.458487 19487 ts_manager.cc:194] Registered new tserver with Master: fbab88bf06264427baefc814d0d5e876 (127.0.98.3:43937)
I20260620 09:20:18.458822 19487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59265
I20260620 09:20:18.461221 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:20:18.468118 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.462470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.0.98.62:43185)
0620 09:20:18.462513 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.462515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.462609 (+ 94us) server_negotiation.cc:407] Connection header received
0620 09:20:18.462670 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.462671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.462680 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.462730 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:18.463239 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.463769 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.464457 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.464659 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.464711 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.465813 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:18.465814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:18.465815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:18.465822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:18.467044 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:18.467341 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:18.467342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:18.467343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:18.467397 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:18.467558 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:18.467558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:18.467559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:18.467751 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:18.467843 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.467940 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.468019 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 09:20:18.469524 19487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:20:18.471024 19487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:20:18.476902 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.473180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.0.98.3:43937)
0620 09:20:18.473351 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.473356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.473365 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:20:18.473401 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.473403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.473417 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.473484 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:18.473645 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.474737 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.475829 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.476518 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.476594 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.476654 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.476780 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.476819 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260620 09:20:18.476907 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.472954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.0.1:60164)
0620 09:20:18.473102 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:18.473140 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:18.473200 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:18.473705 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:18.473706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:18.473709 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:18.473763 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.473765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.474656 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.474657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.475848 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.475851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.476679 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.476680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.476734 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.476737 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:18.476767 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 09:20:18.476832 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":42,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260620 09:20:18.477012 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.473364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.0.98.2:45001)
0620 09:20:18.473515 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.473519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.473529 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:20:18.473565 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.473567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.473600 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.473835 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:18.473905 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.474562 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.475931 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.476591 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.476809 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.476868 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.476890 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.476936 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260620 09:20:18.477058 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.472954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43937 (local address 127.0.0.1:36780)
0620 09:20:18.473303 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:18.473310 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:18.473343 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:18.473493 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:18.473496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:18.473499 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:18.473562 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.473564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.474845 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.474845 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.475754 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.475757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.476602 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.476603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.476678 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.476694 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 09:20:18.476758 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 09:20:18.477000 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":2,"thread_start_us":50,"threads_started":1}
I20260620 09:20:18.476992 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.472946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34921 (local address 127.0.0.1:46236)
0620 09:20:18.473102 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:18.473140 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:18.473200 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:18.473796 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:18.473797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:18.473799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:18.473838 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.473839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.475062 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.475063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.476079 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.476080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.476811 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.476812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.476863 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.476866 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:18.476907 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 09:20:18.476939 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":55,"threads_started":1}
I20260620 09:20:18.477155 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.473205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.0.98.1:34921)
0620 09:20:18.473627 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.473631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.473640 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:20:18.473677 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.473679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.473693 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.473904 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:18.473982 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.474970 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.476146 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.476661 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.476726 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.476782 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.477043 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.477085 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":71,"threads_started":1}
I20260620 09:20:18.477690 19747 tablet_service.cc:1467] Processing CreateTablet for tablet 9e803e73978240929bf3f4e04c090aa3 (DEFAULT_TABLE table=test-table [id=369ade5ce8a64f5f87b263417d9af34b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:18.478062 19747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e803e73978240929bf3f4e04c090aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:18.478142 19613 tablet_service.cc:1467] Processing CreateTablet for tablet 9e803e73978240929bf3f4e04c090aa3 (DEFAULT_TABLE table=test-table [id=369ade5ce8a64f5f87b263417d9af34b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:18.478192 19881 tablet_service.cc:1467] Processing CreateTablet for tablet 9e803e73978240929bf3f4e04c090aa3 (DEFAULT_TABLE table=test-table [id=369ade5ce8a64f5f87b263417d9af34b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:18.478447 19613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e803e73978240929bf3f4e04c090aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:18.478480 19881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e803e73978240929bf3f4e04c090aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:18.481106 19961 tablet_bootstrap.cc:492] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d: Bootstrap starting.
I20260620 09:20:18.481575 19963 tablet_bootstrap.cc:492] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876: Bootstrap starting.
I20260620 09:20:18.481544 19962 tablet_bootstrap.cc:492] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c: Bootstrap starting.
I20260620 09:20:18.482129 19961 tablet_bootstrap.cc:654] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:18.482388 19963 tablet_bootstrap.cc:654] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:18.482515 19962 tablet_bootstrap.cc:654] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:18.482868 19961 log.cc:824] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:18.482910 19963 log.cc:824] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:18.482995 19962 log.cc:824] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:18.483651 19961 tablet_bootstrap.cc:492] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d: No bootstrap required, opened a new log
I20260620 09:20:18.483690 19962 tablet_bootstrap.cc:492] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c: No bootstrap required, opened a new log
I20260620 09:20:18.483721 19961 ts_tablet_manager.cc:1397] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:20:18.483738 19962 ts_tablet_manager.cc:1397] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:20:18.483779 19963 tablet_bootstrap.cc:492] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876: No bootstrap required, opened a new log
I20260620 09:20:18.483824 19963 ts_tablet_manager.cc:1397] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:20:18.485105 19963 raft_consensus.cc:348] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe940c85c07b455185b6961bbfc7640c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "fbab88bf06264427baefc814d0d5e876" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43937 } }
I20260620 09:20:18.485236 19963 raft_consensus.cc:374] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:18.485255 19963 raft_consensus.cc:729] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbab88bf06264427baefc814d0d5e876, State: Initialized, Role: FOLLOWER
I20260620 09:20:18.485352 19963 consensus_queue.cc:260] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe940c85c07b455185b6961bbfc7640c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "fbab88bf06264427baefc814d0d5e876" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43937 } }
I20260620 09:20:18.485596 19963 ts_tablet_manager.cc:1428] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 09:20:18.485638 19947 heartbeater.cc:503] Master 127.0.98.62:43185 was elected leader, sending a full tablet report...
I20260620 09:20:18.485653 19961 raft_consensus.cc:348] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe940c85c07b455185b6961bbfc7640c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "fbab88bf06264427baefc814d0d5e876" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43937 } }
I20260620 09:20:18.485807 19961 raft_consensus.cc:374] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:18.485841 19961 raft_consensus.cc:729] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfbbda84f1f48a4b664c29f6003ef6d, State: Initialized, Role: FOLLOWER
I20260620 09:20:18.485954 19961 consensus_queue.cc:260] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe940c85c07b455185b6961bbfc7640c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "fbab88bf06264427baefc814d0d5e876" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43937 } }
I20260620 09:20:18.486045 19962 raft_consensus.cc:348] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe940c85c07b455185b6961bbfc7640c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "fbab88bf06264427baefc814d0d5e876" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43937 } }
I20260620 09:20:18.486227 19961 ts_tablet_manager.cc:1428] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:20:18.486244 19679 heartbeater.cc:503] Master 127.0.98.62:43185 was elected leader, sending a full tablet report...
I20260620 09:20:18.486198 19962 raft_consensus.cc:374] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:18.486488 19962 raft_consensus.cc:729] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe940c85c07b455185b6961bbfc7640c, State: Initialized, Role: FOLLOWER
I20260620 09:20:18.486617 19962 consensus_queue.cc:260] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe940c85c07b455185b6961bbfc7640c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "fbab88bf06264427baefc814d0d5e876" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43937 } }
I20260620 09:20:18.486867 19962 ts_tablet_manager.cc:1428] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:20:18.486964 19813 heartbeater.cc:503] Master 127.0.98.62:43185 was elected leader, sending a full tablet report...
I20260620 09:20:18.505052 19969 raft_consensus.cc:482] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:20:18.505164 19969 raft_consensus.cc:504] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe940c85c07b455185b6961bbfc7640c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "fbab88bf06264427baefc814d0d5e876" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43937 } }
I20260620 09:20:18.505513 19969 leader_election.cc:288] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cfbbda84f1f48a4b664c29f6003ef6d (127.0.98.1:34921), fbab88bf06264427baefc814d0d5e876 (127.0.98.3:43937)
I20260620 09:20:18.508265 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.505710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36569 (local address 127.0.98.3:43937)
0620 09:20:18.505767 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.505769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.505804 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:20:18.505829 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.505829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.505837 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.505880 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:18.506136 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.506647 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.507540 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.507963 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.508024 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.508035 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.508165 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.508197 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:20:18.508296 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.505647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43937 (local address 127.0.98.2:36569)
0620 09:20:18.505711 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:18.505721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:18.505762 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:18.505895 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:18.505897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:18.505899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:18.505965 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.505967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.506720 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.506721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.507460 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.507463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.508029 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.508030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.508108 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.508111 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:18.508155 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 09:20:18.508197 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 09:20:18.508301 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.505703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42781 (local address 127.0.98.1:34921)
0620 09:20:18.505766 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.505768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.505970 (+ 202us) server_negotiation.cc:407] Connection header received
0620 09:20:18.506033 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.506034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.506046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.506089 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:18.506245 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.506698 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.507592 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.507984 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.508043 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.508052 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.508192 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.508229 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 09:20:18.508426 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.505647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34921 (local address 127.0.98.2:42781)
0620 09:20:18.505896 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:18.505904 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:18.505970 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:18.506100 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:18.506100 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:18.506102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:18.506173 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.506175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.506773 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.506773 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.507517 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.507519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.508050 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.508050 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.508136 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.508139 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:18.508183 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 09:20:18.508231 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":91,"thread_start_us":104,"threads_started":1}
I20260620 09:20:18.508708 19633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e803e73978240929bf3f4e04c090aa3" candidate_uuid: "fe940c85c07b455185b6961bbfc7640c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d" is_pre_election: true
I20260620 09:20:18.508721 19901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e803e73978240929bf3f4e04c090aa3" candidate_uuid: "fe940c85c07b455185b6961bbfc7640c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbab88bf06264427baefc814d0d5e876" is_pre_election: true
I20260620 09:20:18.508842 19633 raft_consensus.cc:2454] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe940c85c07b455185b6961bbfc7640c in term 0.
I20260620 09:20:18.508847 19901 raft_consensus.cc:2454] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe940c85c07b455185b6961bbfc7640c in term 0.
I20260620 09:20:18.509042 19700 leader_election.cc:302] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fbab88bf06264427baefc814d0d5e876, fe940c85c07b455185b6961bbfc7640c; no voters:
I20260620 09:20:18.509204 19969 raft_consensus.cc:2781] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:20:18.509248 19969 raft_consensus.cc:482] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:20:18.509262 19969 raft_consensus.cc:3037] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:18.509883 19969 raft_consensus.cc:504] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe940c85c07b455185b6961bbfc7640c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "fbab88bf06264427baefc814d0d5e876" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43937 } }
I20260620 09:20:18.509997 19969 leader_election.cc:288] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [CANDIDATE]: Term 1 election: Requested vote from peers 5cfbbda84f1f48a4b664c29f6003ef6d (127.0.98.1:34921), fbab88bf06264427baefc814d0d5e876 (127.0.98.3:43937)
I20260620 09:20:18.510144 19633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e803e73978240929bf3f4e04c090aa3" candidate_uuid: "fe940c85c07b455185b6961bbfc7640c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d"
I20260620 09:20:18.510164 19901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e803e73978240929bf3f4e04c090aa3" candidate_uuid: "fe940c85c07b455185b6961bbfc7640c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbab88bf06264427baefc814d0d5e876"
I20260620 09:20:18.510219 19633 raft_consensus.cc:3037] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:18.510232 19901 raft_consensus.cc:3037] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:18.511014 19901 raft_consensus.cc:2454] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe940c85c07b455185b6961bbfc7640c in term 1.
I20260620 09:20:18.511014 19633 raft_consensus.cc:2454] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe940c85c07b455185b6961bbfc7640c in term 1.
I20260620 09:20:18.511181 19703 leader_election.cc:302] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfbbda84f1f48a4b664c29f6003ef6d, fe940c85c07b455185b6961bbfc7640c; no voters:
I20260620 09:20:18.511269 19969 raft_consensus.cc:2781] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:18.511391 19969 raft_consensus.cc:686] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 1 LEADER]: Becoming Leader. State: Replica: fe940c85c07b455185b6961bbfc7640c, State: Running, Role: LEADER
I20260620 09:20:18.511469 19969 consensus_queue.cc:237] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe940c85c07b455185b6961bbfc7640c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } } peers { permanent_uuid: "fbab88bf06264427baefc814d0d5e876" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43937 } }
I20260620 09:20:18.512246 19486 catalog_manager.cc:5495] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe940c85c07b455185b6961bbfc7640c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fe940c85c07b455185b6961bbfc7640c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe940c85c07b455185b6961bbfc7640c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbab88bf06264427baefc814d0d5e876" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260620 09:20:18.517262 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.515323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.0.98.2:45001)
0620 09:20:18.515362 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.515363 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.515370 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:20:18.515442 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.515443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.515457 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.515509 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:18.515588 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.516066 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.516723 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.516888 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.516932 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.517022 (+ 90us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:18.517106 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:18.517157 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.517170 (+ 13us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.517184 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:20:18.518750 19633 raft_consensus.cc:1261] T 9e803e73978240929bf3f4e04c090aa3 P 5cfbbda84f1f48a4b664c29f6003ef6d [term 1 FOLLOWER]: Refusing update from remote peer fe940c85c07b455185b6961bbfc7640c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:18.518750 19901 raft_consensus.cc:1261] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876 [term 1 FOLLOWER]: Refusing update from remote peer fe940c85c07b455185b6961bbfc7640c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:18.519043 19971 consensus_queue.cc:1035] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cfbbda84f1f48a4b664c29f6003ef6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:18.519174 19969 consensus_queue.cc:1035] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbab88bf06264427baefc814d0d5e876" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:18.521317 19976 mvcc.cc:205] Tried to move back new op lower bound from 7298855807050760192 to 7298855807023747072. Current Snapshot: MvccSnapshot[applied={T|T < 7298855807050760192}]
I20260620 09:20:18.521329 19975 mvcc.cc:205] Tried to move back new op lower bound from 7298855807050760192 to 7298855807023747072. Current Snapshot: MvccSnapshot[applied={T|T < 7298855807050760192}]
I20260620 09:20:18.521714 19974 mvcc.cc:205] Tried to move back new op lower bound from 7298855807050760192 to 7298855807023747072. Current Snapshot: MvccSnapshot[applied={T|T < 7298855807050760192}]
I20260620 09:20:18.523599 19487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:20:18.523712 19487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:20:18.524437 19487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1d58dd466fa490796b11534505135d6: Sending DeleteTablet for 3 replicas of tablet 9e803e73978240929bf3f4e04c090aa3
I20260620 09:20:18.524712 19747 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e803e73978240929bf3f4e04c090aa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60164
I20260620 09:20:18.524791 19881 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e803e73978240929bf3f4e04c090aa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36780
I20260620 09:20:18.524938 19984 tablet_replica.cc:330] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c: stopping tablet replica
I20260620 09:20:18.524999 19613 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e803e73978240929bf3f4e04c090aa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46236
I20260620 09:20:18.525133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 19550
I20260620 09:20:18.525992 19984 raft_consensus.cc:2229] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:20:18.526177 19984 raft_consensus.cc:2258] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:18.526217 19985 tablet_replica.cc:330] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876: stopping tablet replica
I20260620 09:20:18.526412 19985 raft_consensus.cc:2229] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:20:18.526572 19985 raft_consensus.cc:2258] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:18.526636 19984 ts_tablet_manager.cc:1905] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:18.526968 19985 ts_tablet_manager.cc:1905] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:18.528347 19984 ts_tablet_manager.cc:1918] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:20:18.528431 19985 ts_tablet_manager.cc:1918] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:20:18.528475 19984 log.cc:1196] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-1/wal/wals/9e803e73978240929bf3f4e04c090aa3
I20260620 09:20:18.528525 19985 log.cc:1196] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/9e803e73978240929bf3f4e04c090aa3
I20260620 09:20:18.528784 19984 ts_tablet_manager.cc:1939] T 9e803e73978240929bf3f4e04c090aa3 P fe940c85c07b455185b6961bbfc7640c: Deleting consensus metadata
I20260620 09:20:18.528839 19985 ts_tablet_manager.cc:1939] T 9e803e73978240929bf3f4e04c090aa3 P fbab88bf06264427baefc814d0d5e876: Deleting consensus metadata
I20260620 09:20:18.529201 19472 catalog_manager.cc:4841] TS fbab88bf06264427baefc814d0d5e876 (127.0.98.3:43937): tablet 9e803e73978240929bf3f4e04c090aa3 (table test-table [id=369ade5ce8a64f5f87b263417d9af34b]) successfully deleted
I20260620 09:20:18.531683 19473 catalog_manager.cc:4841] TS fe940c85c07b455185b6961bbfc7640c (127.0.98.2:45001): tablet 9e803e73978240929bf3f4e04c090aa3 (table test-table [id=369ade5ce8a64f5f87b263417d9af34b]) successfully deleted
W20260620 09:20:18.531749 19475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34921 (error 108)
I20260620 09:20:18.532181 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.532071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34921 (local address 127.0.0.1:46246)
0620 09:20:18.532119 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:18.532153 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 09:20:18.532243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 19684
W20260620 09:20:18.532349 19475 catalog_manager.cc:4584] TS 5cfbbda84f1f48a4b664c29f6003ef6d (127.0.98.1:34921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e803e73978240929bf3f4e04c090aa3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34921: connect: Connection refused (error 111)
I20260620 09:20:18.536990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 19818
I20260620 09:20:18.540939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 19456
2026-06-20T09:20:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1737 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T09:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:18Z Disabled control of system clock
I20260620 09:20:18.565465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43221
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:20:18.642490 19991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:18.642696 19991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:18.642714 19991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:18.643869 19991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:18.643905 19991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:18.643916 19991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:18.643929 19991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:18.643939 19991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:18.645218 19991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:18.645399 19991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:18.645625 19991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:18.650055 19999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:18.650115 19997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:18.650126 19996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:18.650270 19991 server_base.cc:1034] running on GCE node
I20260620 09:20:18.650484 19991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:18.650712 19991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:18.651854 19991 hybrid_clock.cc:648] HybridClock initialized: now 1781947218651837 us; error 28 us; skew 500 ppm
I20260620 09:20:18.653020 19991 webserver.cc:460] Webserver started at http://127.0.98.62:46027/ using document root <none> and password file <none>
I20260620 09:20:18.653256 19991 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:18.653306 19991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:18.653421 19991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:18.654412 19991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "e1ed4a2c55034447befff58f1f54f059"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "3dba5c0514020f36b02886a544b9b0cc"
server_key_iv: "a33bd999148e6698390f949f24ad8680"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.654732 19991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "e1ed4a2c55034447befff58f1f54f059"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "3dba5c0514020f36b02886a544b9b0cc"
server_key_iv: "a33bd999148e6698390f949f24ad8680"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.655879 19991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:20:18.656486 20005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.656629 19991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:20:18.656706 19991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "e1ed4a2c55034447befff58f1f54f059"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "3dba5c0514020f36b02886a544b9b0cc"
server_key_iv: "a33bd999148e6698390f949f24ad8680"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.656764 19991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:18.664448 19991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:18.664702 19991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:18.664819 19991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:18.668399 19991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43221
I20260620 09:20:18.668429 20057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43221 every 8 connection(s)
I20260620 09:20:18.668716 19991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:18.669073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 19991
I20260620 09:20:18.669162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:18.669334 392 external_mini_cluster.cc:1428] Setting key 1790762f3e28251c9a02ac8f6e939ae6
I20260620 09:20:18.669672 20058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:18.672806 20058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059: Bootstrap starting.
I20260620 09:20:18.673365 20058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:18.673636 20058 log.cc:824] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:18.674319 20058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059: No bootstrap required, opened a new log
I20260620 09:20:18.675161 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.669802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.0.98.62:43221)
0620 09:20:18.670013 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.670021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.670050 (+ 29us) server_negotiation.cc:407] Connection header received
0620 09:20:18.670724 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.670733 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.670816 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.670963 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:18.671128 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.672439 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.673235 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.673821 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.673936 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.674604 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:18.674608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:18.674608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:18.674763 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:18.674795 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:18.674805 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:18.674854 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.674936 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.675008 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260620 09:20:18.676258 20058 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ed4a2c55034447befff58f1f54f059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43221 } }
I20260620 09:20:18.676388 20058 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:18.676419 20058 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ed4a2c55034447befff58f1f54f059, State: Initialized, Role: FOLLOWER
I20260620 09:20:18.676524 20058 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ed4a2c55034447befff58f1f54f059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43221 } }
I20260620 09:20:18.676584 20058 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:18.676611 20058 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:18.676656 20058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:18.677186 20058 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ed4a2c55034447befff58f1f54f059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43221 } }
I20260620 09:20:18.677278 20058 leader_election.cc:302] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1ed4a2c55034447befff58f1f54f059; no voters:
I20260620 09:20:18.677441 20058 leader_election.cc:288] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:18.677475 20063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:18.677597 20063 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [term 1 LEADER]: Becoming Leader. State: Replica: e1ed4a2c55034447befff58f1f54f059, State: Running, Role: LEADER
I20260620 09:20:18.677671 20058 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:18.677731 20063 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ed4a2c55034447befff58f1f54f059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43221 } }
I20260620 09:20:18.678102 20064 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1ed4a2c55034447befff58f1f54f059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ed4a2c55034447befff58f1f54f059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43221 } } }
I20260620 09:20:18.678260 20064 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:18.678475 20072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:18.678812 20065 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ed4a2c55034447befff58f1f54f059. Latest consensus state: current_term: 1 leader_uuid: "e1ed4a2c55034447befff58f1f54f059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ed4a2c55034447befff58f1f54f059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43221 } } }
I20260620 09:20:18.678907 20065 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:18.678931 20072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:18.680245 20072 catalog_manager.cc:1347] Generated new cluster ID: e7dbf16bba4b459287c61504664b654b
I20260620 09:20:18.680297 20072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:18.685828 20072 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:18.686335 20072 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:18.693324 20072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059: Generated new TSK 0
I20260620 09:20:18.693466 20072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:18.698662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43221
--builtin_ntp_servers=127.0.98.20:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:20:18.778527 20082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:18.778754 20082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:18.778784 20082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:18.780082 20082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:18.780122 20082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:18.780157 20082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:20:18.781528 20082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:18.781775 20082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:18.782051 20082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:20:18.786747 20082 server_base.cc:1034] running on GCE node
W20260620 09:20:18.786720 20087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:18.786710 20088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:18.786725 20090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:18.787201 20082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:18.787437 20082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:18.788583 20082 hybrid_clock.cc:648] HybridClock initialized: now 1781947218788555 us; error 44 us; skew 500 ppm
I20260620 09:20:18.789775 20082 webserver.cc:460] Webserver started at http://127.0.98.1:42835/ using document root <none> and password file <none>
I20260620 09:20:18.789983 20082 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:18.790031 20082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:18.790180 20082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:18.791203 20082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/data/instance:
uuid: "18003c7083d0424183face31cfa0a528"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "cb3832f269862a61be41babb008f8bdf"
server_key_iv: "2f47da3a07026d5b2d850f4fca1085f6"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.791513 20082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/wal/instance:
uuid: "18003c7083d0424183face31cfa0a528"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "cb3832f269862a61be41babb008f8bdf"
server_key_iv: "2f47da3a07026d5b2d850f4fca1085f6"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.792760 20082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:20:18.793483 20096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.793624 20082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:20:18.793702 20082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/wal
uuid: "18003c7083d0424183face31cfa0a528"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "cb3832f269862a61be41babb008f8bdf"
server_key_iv: "2f47da3a07026d5b2d850f4fca1085f6"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.793761 20082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:18.808048 20082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:18.808355 20082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:18.808494 20082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:18.808718 20082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:18.809055 20082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:18.809094 20082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.809127 20082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:18.809151 20082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.815136 20082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37025
I20260620 09:20:18.815186 20209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37025 every 8 connection(s)
I20260620 09:20:18.815500 20082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:20:18.820322 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.816191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44699 (local address 127.0.98.62:43221)
0620 09:20:18.816246 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.816248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.816472 (+ 224us) server_negotiation.cc:407] Connection header received
0620 09:20:18.816970 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.816971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.816986 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.817038 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:18.817658 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.818159 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.819084 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.819276 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.819331 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.820070 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:18.820072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:18.820072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:18.820076 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:18.820092 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:18.820096 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:18.820145 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.820216 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.820240 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:20:18.820490 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.816183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43221 (local address 127.0.98.1:44699)
0620 09:20:18.816357 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:18.816389 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:18.816872 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:18.817101 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:18.817104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:18.817112 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:18.817570 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.817574 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.818239 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.818240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.818999 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.819002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.819344 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.819344 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.819466 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.819945 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:20:18.819946 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:18.819972 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:20:18.819994 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:18.820146 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:18.820150 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:18.820153 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:18.820213 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:20:18.820310 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260620 09:20:18.820914 20210 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43221
I20260620 09:20:18.820990 20210 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:18.821128 20210 heartbeater.cc:511] Master 127.0.98.62:43221 requested a full tablet report, sending...
I20260620 09:20:18.821549 20022 ts_manager.cc:194] Registered new tserver with Master: 18003c7083d0424183face31cfa0a528 (127.0.98.1:37025)
I20260620 09:20:18.822121 20022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44699
I20260620 09:20:18.823408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 20082
I20260620 09:20:18.823491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:20:18.823702 392 external_mini_cluster.cc:1428] Setting key e11218d843ac004b946b90912aa5a1f5
I20260620 09:20:18.824465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43221
--builtin_ntp_servers=127.0.98.20:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:20:18.903251 20213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:18.903460 20213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:18.903481 20213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:18.904677 20213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:18.904726 20213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:18.904750 20213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:20:18.906327 20213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:18.906570 20213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:18.906953 20213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:18.912569 20218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:18.912539 20213 server_base.cc:1034] running on GCE node
W20260620 09:20:18.912509 20219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:18.912509 20221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:18.912914 20213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:18.913136 20213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:18.914280 20213 hybrid_clock.cc:648] HybridClock initialized: now 1781947218914264 us; error 31 us; skew 500 ppm
I20260620 09:20:18.915509 20213 webserver.cc:460] Webserver started at http://127.0.98.2:44795/ using document root <none> and password file <none>
I20260620 09:20:18.915776 20213 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:18.915835 20213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:18.915959 20213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:18.916993 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/data/instance:
uuid: "c83238bdf98b4425aed246db9b880c98"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "06e434988ee81647c7f1ffd2f4fe282d"
server_key_iv: "e60945355a8462969d01913d1cd1672d"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.917306 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c83238bdf98b4425aed246db9b880c98"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "06e434988ee81647c7f1ffd2f4fe282d"
server_key_iv: "e60945355a8462969d01913d1cd1672d"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.918525 20213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:18.919170 20227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.919304 20213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:18.919382 20213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/wal
uuid: "c83238bdf98b4425aed246db9b880c98"
format_stamp: "Formatted at 2026-06-20 09:20:18 on dist-test-slave-rq8c"
server_key: "06e434988ee81647c7f1ffd2f4fe282d"
server_key_iv: "e60945355a8462969d01913d1cd1672d"
server_key_version: "encryptionkey@0"
I20260620 09:20:18.919445 20213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:18.939123 20213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:18.939414 20213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:18.939534 20213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:18.939771 20213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:18.940078 20213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:18.940109 20213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.940140 20213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:18.940160 20213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:18.945308 20213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41069
I20260620 09:20:18.945358 20340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41069 every 8 connection(s)
I20260620 09:20:18.945648 20213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:20:18.948836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 20213
I20260620 09:20:18.948940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:20:18.949137 392 external_mini_cluster.cc:1428] Setting key 2cce1eb2a4c23c6deddbd5f8ded40207
I20260620 09:20:18.950012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43221
--builtin_ntp_servers=127.0.98.20:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:20:18.951014 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.946298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48997 (local address 127.0.98.62:43221)
0620 09:20:18.946350 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:18.946353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:18.946545 (+ 192us) server_negotiation.cc:407] Connection header received
0620 09:20:18.947051 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:18.947053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:18.947073 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:18.947124 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:18.947831 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.948332 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.949630 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.949881 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.949939 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.950770 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:18.950771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:18.950772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:18.950776 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:18.950791 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:18.950795 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:18.950839 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:18.950911 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 09:20:18.950936 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:20:18.951224 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:18.946270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43221 (local address 127.0.98.2:48997)
0620 09:20:18.946430 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:18.946463 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:18.946954 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:18.947190 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:18.947194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:18.947204 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:18.947746 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.947751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.948414 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.948415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.949546 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:18.949549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:18.949955 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:18.949956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:18.950092 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:18.950639 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:20:18.950640 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:18.950668 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:20:18.950691 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:18.950848 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:18.950853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:18.950856 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:18.950915 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:20:18.951016 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260620 09:20:18.951745 20341 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43221
I20260620 09:20:18.951839 20341 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:18.952018 20341 heartbeater.cc:511] Master 127.0.98.62:43221 requested a full tablet report, sending...
I20260620 09:20:18.952375 20022 ts_manager.cc:194] Registered new tserver with Master: c83238bdf98b4425aed246db9b880c98 (127.0.98.2:41069)
I20260620 09:20:18.952736 20022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48997
W20260620 09:20:19.027757 20344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:19.027940 20344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:19.027959 20344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:19.029074 20344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:19.029112 20344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:19.029139 20344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:20:19.030434 20344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:19.030640 20344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:19.030900 20344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:19.035516 20350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:19.035504 20352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:19.035575 20344 server_base.cc:1034] running on GCE node
W20260620 09:20:19.035511 20349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:19.035880 20344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:19.036069 20344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:19.037194 20344 hybrid_clock.cc:648] HybridClock initialized: now 1781947219037184 us; error 25 us; skew 500 ppm
I20260620 09:20:19.038314 20344 webserver.cc:460] Webserver started at http://127.0.98.3:46013/ using document root <none> and password file <none>
I20260620 09:20:19.038525 20344 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:19.038568 20344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:19.038690 20344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:19.039708 20344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/data/instance:
uuid: "1e28b96049254c3e8e2ff427f325453a"
format_stamp: "Formatted at 2026-06-20 09:20:19 on dist-test-slave-rq8c"
server_key: "752a937473f061c4e48e6de2eecaf68f"
server_key_iv: "034ed1d7a5b85b5c6a73fed29069e0d0"
server_key_version: "encryptionkey@0"
I20260620 09:20:19.040009 20344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1e28b96049254c3e8e2ff427f325453a"
format_stamp: "Formatted at 2026-06-20 09:20:19 on dist-test-slave-rq8c"
server_key: "752a937473f061c4e48e6de2eecaf68f"
server_key_iv: "034ed1d7a5b85b5c6a73fed29069e0d0"
server_key_version: "encryptionkey@0"
I20260620 09:20:19.041100 20344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:20:19.041680 20358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:19.041824 20344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:19.041900 20344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/wal
uuid: "1e28b96049254c3e8e2ff427f325453a"
format_stamp: "Formatted at 2026-06-20 09:20:19 on dist-test-slave-rq8c"
server_key: "752a937473f061c4e48e6de2eecaf68f"
server_key_iv: "034ed1d7a5b85b5c6a73fed29069e0d0"
server_key_version: "encryptionkey@0"
I20260620 09:20:19.041951 20344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:19.075960 20344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:19.076234 20344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:19.076329 20344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:19.076505 20344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:20:19.076771 20344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:20:19.076797 20344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:19.076818 20344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:20:19.076835 20344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:19.082895 20344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35671
I20260620 09:20:19.082970 20471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35671 every 8 connection(s)
I20260620 09:20:19.083258 20344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:20:19.084865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 20344
I20260620 09:20:19.084959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:20:19.085145 392 external_mini_cluster.cc:1428] Setting key 5f00b95e59da4beecea447c8c4e0dca5
I20260620 09:20:19.088402 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.083939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37033 (local address 127.0.98.62:43221)
0620 09:20:19.083988 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:19.084007 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:19.084213 (+ 206us) server_negotiation.cc:407] Connection header received
0620 09:20:19.084819 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:19.084820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:19.084832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:19.084878 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:19.085722 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.086217 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.087133 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.087319 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.087373 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.088151 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:19.088152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:19.088152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:19.088157 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:19.088172 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:19.088176 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:19.088221 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:19.088295 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 09:20:19.088322 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:20:19.088593 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.083924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43221 (local address 127.0.98.3:37033)
0620 09:20:19.084105 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:19.084137 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:19.084711 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:19.084982 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:19.084986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:19.085025 (+ 39us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:19.085639 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:19.085644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.086291 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.086292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:19.087058 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:19.087060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.087383 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.087384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:19.087516 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.088025 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:20:19.088025 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:19.088053 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:20:19.088074 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:19.088227 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:19.088232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:19.088234 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 09:20:19.088293 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:20:19.088393 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260620 09:20:19.088968 20472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43221
I20260620 09:20:19.089048 20472 heartbeater.cc:464] Registering TS with master...
I20260620 09:20:19.089215 20472 heartbeater.cc:511] Master 127.0.98.62:43221 requested a full tablet report, sending...
I20260620 09:20:19.089529 20022 ts_manager.cc:194] Registered new tserver with Master: 1e28b96049254c3e8e2ff427f325453a (127.0.98.3:35671)
I20260620 09:20:19.089875 20022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37033
I20260620 09:20:19.096460 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:20:19.100473 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.097760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54716 (local address 127.0.98.62:43221)
0620 09:20:19.097796 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:19.097798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:19.097903 (+ 105us) server_negotiation.cc:407] Connection header received
0620 09:20:19.097965 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:19.097966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:19.097975 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:19.098026 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:19.098173 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.098653 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.099347 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.099552 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.099604 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.100249 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:19.100250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:19.100250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:19.100255 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:19.100270 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:19.100274 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:19.100310 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:19.100365 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 09:20:19.100386 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:20:19.101893 20022 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:20:19.103374 20022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:20:19.109052 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.105441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41069 (local address 127.0.0.1:48164)
0620 09:20:19.105664 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:19.105683 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:19.105725 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:19.105956 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:19.105956 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:19.105957 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:19.105984 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:19.105985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.106874 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.106875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:19.107986 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:19.107989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.108773 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.108774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:19.108844 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.108857 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 09:20:19.108923 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:20:19.108976 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20260620 09:20:19.109064 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.105645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.0.98.2:41069)
0620 09:20:19.105820 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:19.105824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:19.105833 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:20:19.105867 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:19.105869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:19.105883 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:19.105953 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:19.106030 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.106803 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.108076 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.108705 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.108764 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.108855 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:19.108937 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 09:20:19.108975 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260620 09:20:19.109484 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.105339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.0.98.3:35671)
0620 09:20:19.105631 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:19.105635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:19.105643 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:19.105679 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:19.105681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:19.105697 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:19.105768 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:19.105941 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.107254 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.108475 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.109162 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.109219 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.109295 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:19.109371 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 09:20:19.109410 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":75,"threads_started":1}
I20260620 09:20:19.109303 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.105234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.0.1:34042)
0620 09:20:19.105392 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:19.105434 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:19.105499 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:19.106008 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:19.106009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:19.106012 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:19.106068 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:19.106070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.107369 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.107370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:19.108245 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:19.108247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.109102 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.109103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:19.109162 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.109167 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:20:19.109194 (+ 27us) client_negotiation.cc:240] Negotiation successful
0620 09:20:19.109232 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260620 09:20:19.109655 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.105510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.0.98.1:37025)
0620 09:20:19.105853 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:19.105857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:19.105865 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:19.105898 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:19.105899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:19.105911 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:19.106141 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:19.106206 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.107276 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.108311 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.108985 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.109415 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.109512 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:19.109535 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 09:20:19.109577 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":80,"threads_started":1}
I20260620 09:20:19.109442 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.105243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35671 (local address 127.0.0.1:43694)
0620 09:20:19.105395 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:19.105466 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:19.105505 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:19.105804 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:19.105806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:19.105808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:19.105873 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:19.105875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.107342 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.107343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:19.108398 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:19.108400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.109238 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.109239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:19.109297 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.109300 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:19.109345 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 09:20:19.109382 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260620 09:20:19.110308 20402 tablet_service.cc:1467] Processing CreateTablet for tablet a52298513ad24e17ae9b79dd8c3b6240 (DEFAULT_TABLE table=test-table [id=fc897774c9824643b592c619fe371f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:19.110337 20275 tablet_service.cc:1467] Processing CreateTablet for tablet a52298513ad24e17ae9b79dd8c3b6240 (DEFAULT_TABLE table=test-table [id=fc897774c9824643b592c619fe371f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:19.110658 20402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52298513ad24e17ae9b79dd8c3b6240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:19.110658 20275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52298513ad24e17ae9b79dd8c3b6240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:19.113886 20487 tablet_bootstrap.cc:492] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a: Bootstrap starting.
I20260620 09:20:19.114261 20144 tablet_service.cc:1467] Processing CreateTablet for tablet a52298513ad24e17ae9b79dd8c3b6240 (DEFAULT_TABLE table=test-table [id=fc897774c9824643b592c619fe371f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:20:19.114485 20487 tablet_bootstrap.cc:654] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:19.114563 20144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52298513ad24e17ae9b79dd8c3b6240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:19.114750 20487 log.cc:824] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:19.115028 20486 tablet_bootstrap.cc:492] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98: Bootstrap starting.
I20260620 09:20:19.115430 20487 tablet_bootstrap.cc:492] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a: No bootstrap required, opened a new log
I20260620 09:20:19.115501 20487 ts_tablet_manager.cc:1397] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:20:19.115533 20486 tablet_bootstrap.cc:654] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:19.115820 20486 log.cc:824] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:19.116443 20486 tablet_bootstrap.cc:492] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98: No bootstrap required, opened a new log
I20260620 09:20:19.116504 20486 ts_tablet_manager.cc:1397] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:20:19.117761 20490 tablet_bootstrap.cc:492] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528: Bootstrap starting.
I20260620 09:20:19.117985 20487 raft_consensus.cc:348] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18003c7083d0424183face31cfa0a528" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c83238bdf98b4425aed246db9b880c98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41069 } } peers { permanent_uuid: "1e28b96049254c3e8e2ff427f325453a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35671 } }
I20260620 09:20:19.118120 20487 raft_consensus.cc:374] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:19.118142 20487 raft_consensus.cc:729] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e28b96049254c3e8e2ff427f325453a, State: Initialized, Role: FOLLOWER
I20260620 09:20:19.118275 20487 consensus_queue.cc:260] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18003c7083d0424183face31cfa0a528" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c83238bdf98b4425aed246db9b880c98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41069 } } peers { permanent_uuid: "1e28b96049254c3e8e2ff427f325453a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35671 } }
I20260620 09:20:19.118505 20487 ts_tablet_manager.cc:1428] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:20:19.118515 20490 tablet_bootstrap.cc:654] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:19.118609 20472 heartbeater.cc:503] Master 127.0.98.62:43221 was elected leader, sending a full tablet report...
I20260620 09:20:19.118521 20486 raft_consensus.cc:348] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18003c7083d0424183face31cfa0a528" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c83238bdf98b4425aed246db9b880c98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41069 } } peers { permanent_uuid: "1e28b96049254c3e8e2ff427f325453a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35671 } }
I20260620 09:20:19.118674 20486 raft_consensus.cc:374] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:19.118716 20486 raft_consensus.cc:729] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c83238bdf98b4425aed246db9b880c98, State: Initialized, Role: FOLLOWER
I20260620 09:20:19.118834 20486 consensus_queue.cc:260] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18003c7083d0424183face31cfa0a528" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c83238bdf98b4425aed246db9b880c98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41069 } } peers { permanent_uuid: "1e28b96049254c3e8e2ff427f325453a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35671 } }
I20260620 09:20:19.118974 20490 log.cc:824] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:19.119082 20486 ts_tablet_manager.cc:1428] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 09:20:19.119220 20341 heartbeater.cc:503] Master 127.0.98.62:43221 was elected leader, sending a full tablet report...
I20260620 09:20:19.119794 20490 tablet_bootstrap.cc:492] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528: No bootstrap required, opened a new log
I20260620 09:20:19.119861 20490 ts_tablet_manager.cc:1397] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:20:19.121691 20490 raft_consensus.cc:348] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18003c7083d0424183face31cfa0a528" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c83238bdf98b4425aed246db9b880c98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41069 } } peers { permanent_uuid: "1e28b96049254c3e8e2ff427f325453a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35671 } }
I20260620 09:20:19.121824 20490 raft_consensus.cc:374] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:19.121843 20490 raft_consensus.cc:729] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18003c7083d0424183face31cfa0a528, State: Initialized, Role: FOLLOWER
I20260620 09:20:19.121933 20490 consensus_queue.cc:260] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18003c7083d0424183face31cfa0a528" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c83238bdf98b4425aed246db9b880c98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41069 } } peers { permanent_uuid: "1e28b96049254c3e8e2ff427f325453a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35671 } }
I20260620 09:20:19.122169 20490 ts_tablet_manager.cc:1428] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:20:19.122215 20210 heartbeater.cc:503] Master 127.0.98.62:43221 was elected leader, sending a full tablet report...
W20260620 09:20:19.196161 20342 tablet.cc:2369] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:20:19.307091 20491 raft_consensus.cc:482] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:20:19.307227 20491 raft_consensus.cc:504] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18003c7083d0424183face31cfa0a528" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c83238bdf98b4425aed246db9b880c98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41069 } } peers { permanent_uuid: "1e28b96049254c3e8e2ff427f325453a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35671 } }
I20260620 09:20:19.307576 20491 leader_election.cc:288] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18003c7083d0424183face31cfa0a528 (127.0.98.1:37025), c83238bdf98b4425aed246db9b880c98 (127.0.98.2:41069)
I20260620 09:20:19.310465 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.307871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60745 (local address 127.0.98.1:37025)
0620 09:20:19.307919 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:19.307921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:19.307931 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:20:19.308009 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:19.308010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:19.308024 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:19.308096 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:19.308257 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.308777 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.309820 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.310162 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.310222 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.310232 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:19.310354 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 09:20:19.310393 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 09:20:19.310474 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.308047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36215 (local address 127.0.98.2:41069)
0620 09:20:19.308094 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:19.308096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:19.308106 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:20:19.308186 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:19.308187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:19.308195 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:19.308239 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:20:19.308400 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.308913 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.309813 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.310164 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.310223 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.310233 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:19.310379 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 09:20:19.310413 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:20:19.310472 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.307808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.98.3:60745)
0620 09:20:19.307858 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:19.307870 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:19.307920 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:19.308105 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:19.308106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:19.308108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:19.308182 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:19.308184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.308866 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.308866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:19.309733 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:19.309735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.310238 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.310239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:19.310298 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.310301 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:19.310340 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 09:20:19.310380 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 09:20:19.310487 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.307913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41069 (local address 127.0.98.3:36215)
0620 09:20:19.308063 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:19.308073 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:19.308117 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:19.308257 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:19.308258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:19.308259 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:20:19.308323 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:19.308324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.308984 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.308985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:19.309741 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:19.309743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.310240 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.310241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:19.310316 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.310319 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:20:19.310368 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 09:20:19.310402 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260620 09:20:19.310829 20164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52298513ad24e17ae9b79dd8c3b6240" candidate_uuid: "1e28b96049254c3e8e2ff427f325453a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18003c7083d0424183face31cfa0a528" is_pre_election: true
I20260620 09:20:19.311007 20164 raft_consensus.cc:2454] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e28b96049254c3e8e2ff427f325453a in term 0.
I20260620 09:20:19.310987 20295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52298513ad24e17ae9b79dd8c3b6240" candidate_uuid: "1e28b96049254c3e8e2ff427f325453a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c83238bdf98b4425aed246db9b880c98" is_pre_election: true
I20260620 09:20:19.311105 20295 raft_consensus.cc:2454] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e28b96049254c3e8e2ff427f325453a in term 0.
I20260620 09:20:19.311234 20361 leader_election.cc:302] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18003c7083d0424183face31cfa0a528, 1e28b96049254c3e8e2ff427f325453a; no voters:
I20260620 09:20:19.311343 20491 raft_consensus.cc:2781] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:20:19.311394 20491 raft_consensus.cc:482] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:20:19.311424 20491 raft_consensus.cc:3037] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:19.312103 20491 raft_consensus.cc:504] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18003c7083d0424183face31cfa0a528" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c83238bdf98b4425aed246db9b880c98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41069 } } peers { permanent_uuid: "1e28b96049254c3e8e2ff427f325453a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35671 } }
I20260620 09:20:19.312206 20491 leader_election.cc:288] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [CANDIDATE]: Term 1 election: Requested vote from peers 18003c7083d0424183face31cfa0a528 (127.0.98.1:37025), c83238bdf98b4425aed246db9b880c98 (127.0.98.2:41069)
I20260620 09:20:19.312368 20164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52298513ad24e17ae9b79dd8c3b6240" candidate_uuid: "1e28b96049254c3e8e2ff427f325453a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18003c7083d0424183face31cfa0a528"
I20260620 09:20:19.312379 20295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52298513ad24e17ae9b79dd8c3b6240" candidate_uuid: "1e28b96049254c3e8e2ff427f325453a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c83238bdf98b4425aed246db9b880c98"
I20260620 09:20:19.312445 20164 raft_consensus.cc:3037] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:19.312443 20295 raft_consensus.cc:3037] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:19.313257 20295 raft_consensus.cc:2454] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e28b96049254c3e8e2ff427f325453a in term 1.
I20260620 09:20:19.313256 20164 raft_consensus.cc:2454] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e28b96049254c3e8e2ff427f325453a in term 1.
I20260620 09:20:19.313395 20361 leader_election.cc:302] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18003c7083d0424183face31cfa0a528, 1e28b96049254c3e8e2ff427f325453a; no voters:
I20260620 09:20:19.313478 20491 raft_consensus.cc:2781] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:19.313613 20491 raft_consensus.cc:686] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 1 LEADER]: Becoming Leader. State: Replica: 1e28b96049254c3e8e2ff427f325453a, State: Running, Role: LEADER
I20260620 09:20:19.313720 20491 consensus_queue.cc:237] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18003c7083d0424183face31cfa0a528" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "c83238bdf98b4425aed246db9b880c98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41069 } } peers { permanent_uuid: "1e28b96049254c3e8e2ff427f325453a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35671 } }
I20260620 09:20:19.314512 20021 catalog_manager.cc:5495] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e28b96049254c3e8e2ff427f325453a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1e28b96049254c3e8e2ff427f325453a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18003c7083d0424183face31cfa0a528" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c83238bdf98b4425aed246db9b880c98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e28b96049254c3e8e2ff427f325453a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35671 } health_report { overall_health: HEALTHY } } }
W20260620 09:20:19.316107 20211 tablet.cc:2369] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:20:19.333770 20473 tablet.cc:2369] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:20:19.334129 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.332056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.0.98.3:35671)
0620 09:20:19.332109 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:19.332111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:19.332117 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:20:19.332152 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:19.332152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:19.332169 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:19.332226 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:19.332319 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.332779 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.333447 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:19.333652 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:19.333716 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:19.333813 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:19.333924 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:19.334003 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:19.334019 (+ 16us) server_negotiation.cc:299] Negotiation successful
0620 09:20:19.334041 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:20:19.335752 20164 raft_consensus.cc:1261] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528 [term 1 FOLLOWER]: Refusing update from remote peer 1e28b96049254c3e8e2ff427f325453a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:19.335776 20295 raft_consensus.cc:1261] T a52298513ad24e17ae9b79dd8c3b6240 P c83238bdf98b4425aed246db9b880c98 [term 1 FOLLOWER]: Refusing update from remote peer 1e28b96049254c3e8e2ff427f325453a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:19.336099 20496 consensus_queue.cc:1035] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18003c7083d0424183face31cfa0a528" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:19.336264 20491 consensus_queue.cc:1035] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c83238bdf98b4425aed246db9b880c98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:19.338449 20500 mvcc.cc:205] Tried to move back new op lower bound from 7298855810396753920 to 7298855810309693440. Current Snapshot: MvccSnapshot[applied={T|T < 7298855810396753920}]
I20260620 09:20:19.339180 20501 mvcc.cc:205] Tried to move back new op lower bound from 7298855810396753920 to 7298855810309693440. Current Snapshot: MvccSnapshot[applied={T|T < 7298855810396753920}]
I20260620 09:20:19.340291 20021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:20:19.340373 20021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:20:19.341019 20021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1ed4a2c55034447befff58f1f54f059: Sending DeleteTablet for 3 replicas of tablet a52298513ad24e17ae9b79dd8c3b6240
I20260620 09:20:19.341344 20144 tablet_service.cc:1514] Processing DeleteTablet for tablet a52298513ad24e17ae9b79dd8c3b6240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:19 UTC) from {username='slave'} at 127.0.0.1:34042
I20260620 09:20:19.341348 20275 tablet_service.cc:1514] Processing DeleteTablet for tablet a52298513ad24e17ae9b79dd8c3b6240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:19 UTC) from {username='slave'} at 127.0.0.1:48164
I20260620 09:20:19.341539 20509 tablet_replica.cc:330] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528: stopping tablet replica
I20260620 09:20:19.341518 20402 tablet_service.cc:1514] Processing DeleteTablet for tablet a52298513ad24e17ae9b79dd8c3b6240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:20:19 UTC) from {username='slave'} at 127.0.0.1:43694
I20260620 09:20:19.341630 20509 raft_consensus.cc:2229] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:20:19.341755 20509 raft_consensus.cc:2258] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:19.341769 20511 tablet_replica.cc:330] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a: stopping tablet replica
I20260620 09:20:19.341933 20511 raft_consensus.cc:2229] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:20:19.342080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 20082
I20260620 09:20:19.342224 20511 raft_consensus.cc:2258] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:20:19.342234 20509 ts_tablet_manager.cc:1905] T a52298513ad24e17ae9b79dd8c3b6240 P 18003c7083d0424183face31cfa0a528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:19.342716 20511 ts_tablet_manager.cc:1905] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:20:19.344396 20511 ts_tablet_manager.cc:1918] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:20:19.344523 20511 log.cc:1196] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a: Deleting WAL directory at /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946986834672-392-0/minicluster-data/ts-2/wal/wals/a52298513ad24e17ae9b79dd8c3b6240
I20260620 09:20:19.344817 20511 ts_tablet_manager.cc:1939] T a52298513ad24e17ae9b79dd8c3b6240 P 1e28b96049254c3e8e2ff427f325453a: Deleting consensus metadata
I20260620 09:20:19.345247 20007 catalog_manager.cc:4841] TS 1e28b96049254c3e8e2ff427f325453a (127.0.98.3:35671): tablet a52298513ad24e17ae9b79dd8c3b6240 (table test-table [id=fc897774c9824643b592c619fe371f33]) successfully deleted
W20260620 09:20:19.347491 20008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37025 (error 108)
I20260620 09:20:19.347666 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 20213
I20260620 09:20:19.349115 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.348976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.0.1:34054)
0620 09:20:19.349034 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:19.349082 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 09:20:19.349234 20008 catalog_manager.cc:4584] TS 18003c7083d0424183face31cfa0a528 (127.0.98.1:37025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a52298513ad24e17ae9b79dd8c3b6240: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37025: connect: Connection refused (error 111)
I20260620 09:20:19.351745 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:19.351653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41069 (local address 127.0.0.1:48174)
0620 09:20:19.351686 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:19.351713 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 09:20:19.355744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 20344
I20260620 09:20:19.359797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 19991
2026-06-20T09:20:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (818 ms)
[----------] 6 tests from EncryptionPolicyTest (7743 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:19 dist-test-slave-rq8c krb5kdc[20515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:19 dist-test-slave-rq8c krb5kdc[20515](info): set up 2 sockets
Jun 20 09:20:19 dist-test-slave-rq8c krb5kdc[20515](info): commencing operation
krb5kdc: starting...
W20260620 09:20:20.303269 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:20 dist-test-slave-rq8c krb5kdc[20515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:20:20.409489 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:20.487249 20528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:20.487460 20528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:20.487493 20528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:20.488662 20528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:20.488708 20528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:20.488725 20528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:20.488741 20528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:20.488755 20528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:20.490214 20528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40623
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:20.490458 20528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:20.490658 20528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:20.495193 20533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:20.495373 20536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:20.495223 20528 server_base.cc:1034] running on GCE node
W20260620 09:20:20.495203 20534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:20.495728 20528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:20.495995 20528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:20.497135 20528 hybrid_clock.cc:648] HybridClock initialized: now 1781947220497121 us; error 29 us; skew 500 ppm
Jun 20 09:20:20 dist-test-slave-rq8c krb5kdc[20515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:20.499529 20528 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:20:20.499959 20528 webserver.cc:460] Webserver started at http://127.0.0.1:43283/ using document root <none> and password file <none>
I20260620 09:20:20.500152 20528 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:20.500195 20528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:20.500316 20528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:20.501353 20528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "125a3ea3a81e4d98b67521eebff71b48"
format_stamp: "Formatted at 2026-06-20 09:20:20 on dist-test-slave-rq8c"
server_key: "e821794c47be879ce9fbf2b602b00d1b"
server_key_iv: "3a03af710a1eac1556b4bb659d5ea040"
server_key_version: "encryptionkey@0"
I20260620 09:20:20.501703 20528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "125a3ea3a81e4d98b67521eebff71b48"
format_stamp: "Formatted at 2026-06-20 09:20:20 on dist-test-slave-rq8c"
server_key: "e821794c47be879ce9fbf2b602b00d1b"
server_key_iv: "3a03af710a1eac1556b4bb659d5ea040"
server_key_version: "encryptionkey@0"
I20260620 09:20:20.502935 20528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:20:20.503578 20543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:20.503746 20528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:20:20.503837 20528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "125a3ea3a81e4d98b67521eebff71b48"
format_stamp: "Formatted at 2026-06-20 09:20:20 on dist-test-slave-rq8c"
server_key: "e821794c47be879ce9fbf2b602b00d1b"
server_key_iv: "3a03af710a1eac1556b4bb659d5ea040"
server_key_version: "encryptionkey@0"
I20260620 09:20:20.503896 20528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:20.523999 20528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:20.526074 20528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:20.526185 20528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:20.529796 20528 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40623
I20260620 09:20:20.529841 20595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40623 every 8 connection(s)
I20260620 09:20:20.530143 20528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:20.530961 20596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:20.533186 20596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48: Bootstrap starting.
I20260620 09:20:20.533728 20596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:20.533725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 20528
I20260620 09:20:20.533809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:20.533983 392 external_mini_cluster.cc:1428] Setting key c20b53666d94adb6c3d1d89c289a2731
I20260620 09:20:20.533991 20596 log.cc:824] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:20.534693 20596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48: No bootstrap required, opened a new log
I20260620 09:20:20.536945 20596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125a3ea3a81e4d98b67521eebff71b48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } }
I20260620 09:20:20.537084 20596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:20.537117 20596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125a3ea3a81e4d98b67521eebff71b48, State: Initialized, Role: FOLLOWER
I20260620 09:20:20.537237 20596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125a3ea3a81e4d98b67521eebff71b48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } }
I20260620 09:20:20.537307 20596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:20.537336 20596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:20.537381 20596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:20.537890 20596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125a3ea3a81e4d98b67521eebff71b48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } }
I20260620 09:20:20.537981 20596 leader_election.cc:302] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 125a3ea3a81e4d98b67521eebff71b48; no voters:
I20260620 09:20:20.538141 20596 leader_election.cc:288] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:20.538179 20601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:20.538285 20601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [term 1 LEADER]: Becoming Leader. State: Replica: 125a3ea3a81e4d98b67521eebff71b48, State: Running, Role: LEADER
I20260620 09:20:20.538367 20601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125a3ea3a81e4d98b67521eebff71b48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } }
I20260620 09:20:20.538407 20596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:20.538674 20601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "125a3ea3a81e4d98b67521eebff71b48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125a3ea3a81e4d98b67521eebff71b48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } } }
I20260620 09:20:20.538738 20601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:20.538837 20601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 125a3ea3a81e4d98b67521eebff71b48. Latest consensus state: current_term: 1 leader_uuid: "125a3ea3a81e4d98b67521eebff71b48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125a3ea3a81e4d98b67521eebff71b48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } } }
I20260620 09:20:20.538883 20601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:20.538997 20606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:20.539700 20606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 09:20:20 dist-test-slave-rq8c krb5kdc[20515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:20:20.541671 20606 catalog_manager.cc:1347] Generated new cluster ID: 27db5bc7412e45959e0079292bb95078
I20260620 09:20:20.541750 20606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:20.545323 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:20.534460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60737 (local address 127.0.0.1:40623)
0620 09:20:20.534614 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:20.534621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:20.534646 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:20:20.535174 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:20.535184 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:20.535245 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:20.535389 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:20.536039 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:20.536916 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:20.537775 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:20.538245 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:20.538339 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:20.541303 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:20.541307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:20.541307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:20.541318 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:20.543197 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:20.543674 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:20.543677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:20.543678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:20.543783 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:20.544025 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:20.544026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:20.544026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:20.544345 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:20.544503 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:20.545003 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 09:20:20.545123 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260620 09:20:20.546748 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:20.555094 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:20.548238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38735 (local address 127.0.0.1:40623)
0620 09:20:20.548276 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:20.548277 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:20.548373 (+ 96us) server_negotiation.cc:407] Connection header received
0620 09:20:20.548430 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:20.548430 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:20.548440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:20.548489 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:20.549036 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:20.549784 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:20.550710 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:20.551041 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:20.551110 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:20.552369 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:20.552371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:20.552371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:20.552378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:20.553940 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:20.554333 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:20.554334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:20.554334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:20.554387 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:20.554582 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:20.554583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:20.554583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:20.554705 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:20.554804 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:20.554939 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 09:20:20.555001 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:20:20.555325 20606 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:20.556036 20606 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:20.561642 20606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 125a3ea3a81e4d98b67521eebff71b48: Generated new TSK 0
I20260620 09:20:20.561821 20606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:20.564905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 20528
W20260620 09:20:20.568784 20513 connection.cc:535] client connection to 127.0.0.1:40623 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40623): Connection reset by peer (error 104)
2026-06-20T09:20:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:20 dist-test-slave-rq8c krb5kdc[20627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:20 dist-test-slave-rq8c krb5kdc[20627](info): set up 2 sockets
Jun 20 09:20:20 dist-test-slave-rq8c krb5kdc[20627](info): commencing operation
krb5kdc: starting...
W20260620 09:20:21.507292 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:21 dist-test-slave-rq8c krb5kdc[20627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:20:21.611382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:21.688273 20640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:21.688482 20640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:21.688504 20640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:21.689603 20640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:21.689635 20640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:21.689648 20640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:21.689659 20640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:21.689670 20640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:21.691057 20640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:21.691247 20640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:21.691426 20640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:21.697191 20648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:21.697242 20645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:21.697183 20646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:21.697382 20640 server_base.cc:1034] running on GCE node
I20260620 09:20:21.697630 20640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:21.697887 20640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:21.699044 20640 hybrid_clock.cc:648] HybridClock initialized: now 1781947221699031 us; error 27 us; skew 500 ppm
Jun 20 09:20:21 dist-test-slave-rq8c krb5kdc[20627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:21.704846 20640 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:20:21.705271 20640 webserver.cc:460] Webserver started at http://127.0.0.1:42237/ using document root <none> and password file <none>
I20260620 09:20:21.705466 20640 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:21.705508 20640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:21.705621 20640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:21.706629 20640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "e4cd31b6e6a8410fa77e1c06b00b9b29"
format_stamp: "Formatted at 2026-06-20 09:20:21 on dist-test-slave-rq8c"
server_key: "a17f04aa38a0a8f61b5b141d7729301d"
server_key_iv: "ec506dff043b37ebd97f5488917226f4"
server_key_version: "encryptionkey@0"
I20260620 09:20:21.706952 20640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4cd31b6e6a8410fa77e1c06b00b9b29"
format_stamp: "Formatted at 2026-06-20 09:20:21 on dist-test-slave-rq8c"
server_key: "a17f04aa38a0a8f61b5b141d7729301d"
server_key_iv: "ec506dff043b37ebd97f5488917226f4"
server_key_version: "encryptionkey@0"
I20260620 09:20:21.708236 20640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 09:20:21.708870 20655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:21.709021 20640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:21.709092 20640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "e4cd31b6e6a8410fa77e1c06b00b9b29"
format_stamp: "Formatted at 2026-06-20 09:20:21 on dist-test-slave-rq8c"
server_key: "a17f04aa38a0a8f61b5b141d7729301d"
server_key_iv: "ec506dff043b37ebd97f5488917226f4"
server_key_version: "encryptionkey@0"
I20260620 09:20:21.709156 20640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:21.734583 20640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:21.736719 20640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:21.736845 20640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:21.740530 20640 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33777
I20260620 09:20:21.740563 20707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33777 every 8 connection(s)
I20260620 09:20:21.740873 20640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:21.741717 20708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:21.743899 20708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29: Bootstrap starting.
I20260620 09:20:21.744408 20708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:21.744626 20708 log.cc:824] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:21.745225 20708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29: No bootstrap required, opened a new log
I20260620 09:20:21.745566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 20640
I20260620 09:20:21.745666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:21.745803 392 external_mini_cluster.cc:1428] Setting key 8b552e80128a82dc31713e375d031a37
I20260620 09:20:21.747020 20708 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd31b6e6a8410fa77e1c06b00b9b29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33777 } }
I20260620 09:20:21.747150 20708 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:21.747181 20708 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cd31b6e6a8410fa77e1c06b00b9b29, State: Initialized, Role: FOLLOWER
I20260620 09:20:21.747318 20708 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd31b6e6a8410fa77e1c06b00b9b29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33777 } }
I20260620 09:20:21.747395 20708 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:21.747445 20708 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:21.747495 20708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:21.748044 20708 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd31b6e6a8410fa77e1c06b00b9b29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33777 } }
I20260620 09:20:21.748137 20708 leader_election.cc:302] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4cd31b6e6a8410fa77e1c06b00b9b29; no voters:
I20260620 09:20:21.748287 20708 leader_election.cc:288] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:21.748323 20713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:21.748466 20713 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [term 1 LEADER]: Becoming Leader. State: Replica: e4cd31b6e6a8410fa77e1c06b00b9b29, State: Running, Role: LEADER
I20260620 09:20:21.748531 20708 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:21.748579 20713 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd31b6e6a8410fa77e1c06b00b9b29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33777 } }
I20260620 09:20:21.748932 20714 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4cd31b6e6a8410fa77e1c06b00b9b29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd31b6e6a8410fa77e1c06b00b9b29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33777 } } }
I20260620 09:20:21.749013 20714 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:21.749799 20720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:21.750051 20713 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4cd31b6e6a8410fa77e1c06b00b9b29. Latest consensus state: current_term: 1 leader_uuid: "e4cd31b6e6a8410fa77e1c06b00b9b29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cd31b6e6a8410fa77e1c06b00b9b29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33777 } } }
I20260620 09:20:21.751008 20713 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:21.751320 20720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:21.751771 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:21.746252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46037 (local address 127.0.0.1:33777)
0620 09:20:21.746410 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:21.746417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:21.746441 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:20:21.747021 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:21.747028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:21.747079 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:21.747214 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:21.747426 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:21.748865 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:21.749761 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:21.750295 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:21.750398 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:21.751032 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:21.751036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:21.751036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:21.751239 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:21.751277 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:21.751292 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:21.751373 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:21.751515 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 09:20:21.751544 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260620 09:20:21.752627 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:21.754027 20720 catalog_manager.cc:1347] Generated new cluster ID: 84cd23764fef4aa5b28ac9acef6e0689
I20260620 09:20:21.754071 20720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:21.757375 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:21.753893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54215 (local address 127.0.0.1:33777)
0620 09:20:21.753925 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:21.753927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:21.754167 (+ 240us) server_negotiation.cc:407] Connection header received
0620 09:20:21.754220 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:21.754221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:21.754232 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:21.754277 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:21.754429 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:21.755080 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:21.756074 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:21.756393 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:21.756426 (+ 33us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:21.757132 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:21.757133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:21.757133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:21.757137 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:21.757149 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:21.757153 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:21.757180 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:21.757285 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 09:20:21.757297 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:20:21.760049 20720 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:21.760726 20720 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:21.766799 20720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4cd31b6e6a8410fa77e1c06b00b9b29: Generated new TSK 0
I20260620 09:20:21.766961 20720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:21.771826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 20640
W20260620 09:20:21.775561 20625 connection.cc:535] client connection to 127.0.0.1:33777 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33777): Connection reset by peer (error 104)
2026-06-20T09:20:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:21 dist-test-slave-rq8c krb5kdc[20739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:21 dist-test-slave-rq8c krb5kdc[20739](info): set up 2 sockets
Jun 20 09:20:21 dist-test-slave-rq8c krb5kdc[20739](info): commencing operation
krb5kdc: starting...
W20260620 09:20:22.719290 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:22 dist-test-slave-rq8c krb5kdc[20739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:20:22.831197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:22.909576 20752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:22.909770 20752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:22.909791 20752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:22.910894 20752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:22.910928 20752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:22.910941 20752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:22.910951 20752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:22.910962 20752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:22.912381 20752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37413
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:22.912565 20752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:22.912725 20752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:22.917398 20757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:22.917428 20752 server_base.cc:1034] running on GCE node
W20260620 09:20:22.917435 20760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:22.917397 20758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:22.917826 20752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:22.918156 20752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:22.919322 20752 hybrid_clock.cc:648] HybridClock initialized: now 1781947222919303 us; error 35 us; skew 500 ppm
Jun 20 09:20:22 dist-test-slave-rq8c krb5kdc[20739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:22.921788 20752 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:20:22.922221 20752 webserver.cc:460] Webserver started at http://127.0.0.1:39641/ using document root <none> and password file <none>
I20260620 09:20:22.922436 20752 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:22.922482 20752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:22.922605 20752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:22.923648 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "61a2613235b34b72acf30e26d716bde1"
format_stamp: "Formatted at 2026-06-20 09:20:22 on dist-test-slave-rq8c"
server_key: "1186c73062506242a3c3d0a765f898d7"
server_key_iv: "0e17394a4abbccc3b4d0a29f8715ec8d"
server_key_version: "encryptionkey@0"
I20260620 09:20:22.923981 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "61a2613235b34b72acf30e26d716bde1"
format_stamp: "Formatted at 2026-06-20 09:20:22 on dist-test-slave-rq8c"
server_key: "1186c73062506242a3c3d0a765f898d7"
server_key_iv: "0e17394a4abbccc3b4d0a29f8715ec8d"
server_key_version: "encryptionkey@0"
I20260620 09:20:22.925242 20752 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:20:22.926011 20767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:22.926239 20752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:20:22.926332 20752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "61a2613235b34b72acf30e26d716bde1"
format_stamp: "Formatted at 2026-06-20 09:20:22 on dist-test-slave-rq8c"
server_key: "1186c73062506242a3c3d0a765f898d7"
server_key_iv: "0e17394a4abbccc3b4d0a29f8715ec8d"
server_key_version: "encryptionkey@0"
I20260620 09:20:22.926398 20752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:22.929557 20752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:22.931262 20752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:22.931391 20752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:22.934945 20752 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37413
I20260620 09:20:22.934988 20819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37413 every 8 connection(s)
I20260620 09:20:22.935271 20752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:22.936208 20820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:22.938252 20820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1: Bootstrap starting.
I20260620 09:20:22.938728 20820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:22.938959 20820 log.cc:824] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:22.939487 20820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1: No bootstrap required, opened a new log
I20260620 09:20:22.941025 20820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a2613235b34b72acf30e26d716bde1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37413 } }
I20260620 09:20:22.941154 20820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:22.941174 20820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61a2613235b34b72acf30e26d716bde1, State: Initialized, Role: FOLLOWER
I20260620 09:20:22.941267 20820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a2613235b34b72acf30e26d716bde1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37413 } }
I20260620 09:20:22.941318 20820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:22.941340 20820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:22.941370 20820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:22.941833 20820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a2613235b34b72acf30e26d716bde1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37413 } }
I20260620 09:20:22.941915 20820 leader_election.cc:302] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61a2613235b34b72acf30e26d716bde1; no voters:
I20260620 09:20:22.942028 20820 leader_election.cc:288] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:22.942080 20823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:22.942245 20820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:22.942421 20823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [term 1 LEADER]: Becoming Leader. State: Replica: 61a2613235b34b72acf30e26d716bde1, State: Running, Role: LEADER
I20260620 09:20:22.942508 20823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a2613235b34b72acf30e26d716bde1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37413 } }
I20260620 09:20:22.942839 20824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61a2613235b34b72acf30e26d716bde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a2613235b34b72acf30e26d716bde1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37413 } } }
I20260620 09:20:22.942847 20825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61a2613235b34b72acf30e26d716bde1. Latest consensus state: current_term: 1 leader_uuid: "61a2613235b34b72acf30e26d716bde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a2613235b34b72acf30e26d716bde1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37413 } } }
I20260620 09:20:22.942934 20824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:22.942941 20825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:22.943113 20832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:22.943590 20832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:22.945118 20832 catalog_manager.cc:1347] Generated new cluster ID: d27da10bb7e94b5f86635963e3fbeecb
I20260620 09:20:22.945163 20832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:22.945215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 20752
I20260620 09:20:22.945333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:22.945510 392 external_mini_cluster.cc:1428] Setting key 3baced1a487a486889e9fa8d4fd2b2fd
I20260620 09:20:22.957365 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:22.945930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45601 (local address 127.0.0.1:37413)
0620 09:20:22.946080 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:22.946088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:22.955874 (+ 9786us) server_negotiation.cc:407] Connection header received
0620 09:20:22.956019 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:22.956033 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:22.956065 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:22.956223 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:22.956892 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:22.956896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:22.956896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:22.956901 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:22.956948 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:22.956961 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:22.957025 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:22.957086 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 09:20:22.957146 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260620 09:20:22.958099 20832 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:22.958238 20832 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:22.958381 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:22.960877 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:22.959849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52655 (local address 127.0.0.1:37413)
0620 09:20:22.959882 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:22.959884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:22.960023 (+ 139us) server_negotiation.cc:407] Connection header received
0620 09:20:22.960099 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:22.960099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:22.960104 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:22.960127 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:22.960683 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:22.960684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:22.960684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:22.960687 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:22.960699 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:22.960703 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:22.960727 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:22.960807 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 09:20:22.960818 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:20:22.971422 20832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61a2613235b34b72acf30e26d716bde1: Generated new TSK 0
I20260620 09:20:22.971603 20832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:22.980324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 20752
2026-06-20T09:20:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:23 dist-test-slave-rq8c krb5kdc[20851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:23 dist-test-slave-rq8c krb5kdc[20851](info): set up 2 sockets
Jun 20 09:20:23 dist-test-slave-rq8c krb5kdc[20851](info): commencing operation
krb5kdc: starting...
W20260620 09:20:23.923305 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:23 dist-test-slave-rq8c krb5kdc[20851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:20:24.030530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:24.109565 20864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:24.109766 20864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:24.109797 20864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:24.110934 20864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:24.110973 20864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:24.110992 20864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:24.111007 20864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:24.111024 20864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:24.112519 20864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:24.112741 20864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:24.112933 20864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:24.117408 20869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:24.117607 20872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:24.117607 20870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:24.117695 20864 server_base.cc:1034] running on GCE node
I20260620 09:20:24.117944 20864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:24.118196 20864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:24.119330 20864 hybrid_clock.cc:648] HybridClock initialized: now 1781947224119323 us; error 29 us; skew 500 ppm
Jun 20 09:20:24 dist-test-slave-rq8c krb5kdc[20851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:24.121645 20864 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:20:24.122028 20864 webserver.cc:460] Webserver started at http://127.0.0.1:42661/ using document root <none> and password file <none>
I20260620 09:20:24.122224 20864 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:24.122269 20864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:24.122386 20864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:24.123386 20864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "06a674e6374644f2a6a13e3dd403562e"
format_stamp: "Formatted at 2026-06-20 09:20:24 on dist-test-slave-rq8c"
server_key: "995fadc245516b4c8b6605e7ba2bbad9"
server_key_iv: "932060ed4b3a6cfe7497397ae083bf25"
server_key_version: "encryptionkey@0"
I20260620 09:20:24.123733 20864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "06a674e6374644f2a6a13e3dd403562e"
format_stamp: "Formatted at 2026-06-20 09:20:24 on dist-test-slave-rq8c"
server_key: "995fadc245516b4c8b6605e7ba2bbad9"
server_key_iv: "932060ed4b3a6cfe7497397ae083bf25"
server_key_version: "encryptionkey@0"
I20260620 09:20:24.124962 20864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:20:24.125631 20879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:24.125810 20864 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:20:24.125893 20864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "06a674e6374644f2a6a13e3dd403562e"
format_stamp: "Formatted at 2026-06-20 09:20:24 on dist-test-slave-rq8c"
server_key: "995fadc245516b4c8b6605e7ba2bbad9"
server_key_iv: "932060ed4b3a6cfe7497397ae083bf25"
server_key_version: "encryptionkey@0"
I20260620 09:20:24.125957 20864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:24.148165 20864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:24.150278 20864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:24.150418 20864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:24.154117 20864 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37421
I20260620 09:20:24.154162 20931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37421 every 8 connection(s)
I20260620 09:20:24.154490 20864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:24.155038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 20864
I20260620 09:20:24.155150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:24.155344 392 external_mini_cluster.cc:1428] Setting key b37587e86f7b4166a14c2fcd900190f3
I20260620 09:20:24.155468 20932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:24.158599 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e: Bootstrap starting.
I20260620 09:20:24.159260 20932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:24.159548 20932 log.cc:824] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:24.160233 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e: No bootstrap required, opened a new log
Jun 20 09:20:24 dist-test-slave-rq8c krb5kdc[20851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:20:24.162528 20932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a674e6374644f2a6a13e3dd403562e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37421 } }
I20260620 09:20:24.162693 20932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:24.162737 20932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a674e6374644f2a6a13e3dd403562e, State: Initialized, Role: FOLLOWER
I20260620 09:20:24.162856 20932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a674e6374644f2a6a13e3dd403562e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37421 } }
I20260620 09:20:24.162914 20932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:24.162941 20932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:24.162985 20932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:24.163467 20932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a674e6374644f2a6a13e3dd403562e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37421 } }
I20260620 09:20:24.163551 20932 leader_election.cc:302] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06a674e6374644f2a6a13e3dd403562e; no voters:
I20260620 09:20:24.163722 20932 leader_election.cc:288] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:24.163767 20937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:24.163904 20937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [term 1 LEADER]: Becoming Leader. State: Replica: 06a674e6374644f2a6a13e3dd403562e, State: Running, Role: LEADER
I20260620 09:20:24.163969 20932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:24.164022 20937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a674e6374644f2a6a13e3dd403562e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37421 } }
I20260620 09:20:24.164448 20939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06a674e6374644f2a6a13e3dd403562e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a674e6374644f2a6a13e3dd403562e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37421 } } }
I20260620 09:20:24.164530 20939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:24.164721 20937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a674e6374644f2a6a13e3dd403562e. Latest consensus state: current_term: 1 leader_uuid: "06a674e6374644f2a6a13e3dd403562e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a674e6374644f2a6a13e3dd403562e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37421 } } }
I20260620 09:20:24.164997 20937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:24.165143 20946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:24.165606 20946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:24.166260 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:24.155871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60885 (local address 127.0.0.1:37421)
0620 09:20:24.156035 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:24.156044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:24.156069 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:20:24.156541 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:24.156548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:24.156594 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:24.156704 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:24.157466 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:24.158586 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:24.159580 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:24.160158 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:24.160244 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:24.162661 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:24.162664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:24.162665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:24.162674 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:24.164544 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:24.164948 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:24.164951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:24.164951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:24.165055 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:24.165281 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:24.165282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:24.165282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:24.165636 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:24.165819 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:24.165943 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 09:20:24.166072 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260620 09:20:24.167068 20946 catalog_manager.cc:1347] Generated new cluster ID: 2296eb224aca4bd2a125a398b6634b81
I20260620 09:20:24.167140 20946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:24.167284 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:24.175189 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:24.168739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57645 (local address 127.0.0.1:37421)
0620 09:20:24.168786 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:24.168787 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:24.168817 (+ 30us) server_negotiation.cc:407] Connection header received
0620 09:20:24.168888 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:24.168888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:24.168899 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:24.168948 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:24.169480 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:24.170206 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:24.171131 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:24.171455 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:24.171518 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:24.172744 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:24.172745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:24.172745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:24.172752 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:24.174180 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:24.174520 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:24.174521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:24.174522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:24.174576 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:24.174738 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:24.174738 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:24.174739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:24.174862 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:24.174951 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:24.175036 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 09:20:24.175104 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:20:24.178514 20946 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:24.179010 20946 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:24.182374 20946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06a674e6374644f2a6a13e3dd403562e: Generated new TSK 0
I20260620 09:20:24.182539 20946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:24.188331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 20864
2026-06-20T09:20:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:24 dist-test-slave-rq8c krb5kdc[20963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:24 dist-test-slave-rq8c krb5kdc[20963](info): set up 2 sockets
Jun 20 09:20:24 dist-test-slave-rq8c krb5kdc[20963](info): commencing operation
krb5kdc: starting...
W20260620 09:20:25.139358 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:25 dist-test-slave-rq8c krb5kdc[20963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:20:25.247210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39095
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:25.323868 20976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:25.324079 20976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:25.324110 20976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:25.325286 20976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:25.325327 20976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:25.325345 20976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:25.325361 20976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:25.325377 20976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:25.326895 20976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:25.327142 20976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:25.327342 20976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:25.332039 20981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:25.332032 20984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:25.332057 20982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:25.332297 20976 server_base.cc:1034] running on GCE node
I20260620 09:20:25.332517 20976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:25.332791 20976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:25.333925 20976 hybrid_clock.cc:648] HybridClock initialized: now 1781947225333913 us; error 30 us; skew 500 ppm
Jun 20 09:20:25 dist-test-slave-rq8c krb5kdc[20963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:25.336264 20976 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:20:25.336668 20976 webserver.cc:460] Webserver started at http://127.0.0.1:41587/ using document root <none> and password file <none>
I20260620 09:20:25.336864 20976 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:25.336911 20976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:25.337033 20976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:25.338045 20976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "0ebd331723fb4655837031212c7af0c7"
format_stamp: "Formatted at 2026-06-20 09:20:25 on dist-test-slave-rq8c"
server_key: "eed497c1ac8016b9f97c066a8cde2703"
server_key_iv: "26322799a406aa5b5b44a7dddfdec0bd"
server_key_version: "encryptionkey@0"
I20260620 09:20:25.338371 20976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ebd331723fb4655837031212c7af0c7"
format_stamp: "Formatted at 2026-06-20 09:20:25 on dist-test-slave-rq8c"
server_key: "eed497c1ac8016b9f97c066a8cde2703"
server_key_iv: "26322799a406aa5b5b44a7dddfdec0bd"
server_key_version: "encryptionkey@0"
I20260620 09:20:25.339561 20976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:20:25.340236 20991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:25.340377 20976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:20:25.340456 20976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "0ebd331723fb4655837031212c7af0c7"
format_stamp: "Formatted at 2026-06-20 09:20:25 on dist-test-slave-rq8c"
server_key: "eed497c1ac8016b9f97c066a8cde2703"
server_key_iv: "26322799a406aa5b5b44a7dddfdec0bd"
server_key_version: "encryptionkey@0"
I20260620 09:20:25.340556 20976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:25.359074 20976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:25.360925 20976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:25.361058 20976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:25.364663 20976 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39095
I20260620 09:20:25.364699 21043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39095 every 8 connection(s)
I20260620 09:20:25.365023 20976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:25.365823 21044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:25.367938 21044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7: Bootstrap starting.
I20260620 09:20:25.368419 21044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:25.368631 21044 log.cc:824] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:25.369230 21044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7: No bootstrap required, opened a new log
I20260620 09:20:25.370700 21044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebd331723fb4655837031212c7af0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39095 } }
I20260620 09:20:25.370839 21044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:25.370860 21044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebd331723fb4655837031212c7af0c7, State: Initialized, Role: FOLLOWER
I20260620 09:20:25.370963 21044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebd331723fb4655837031212c7af0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39095 } }
I20260620 09:20:25.371034 21044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:25.371063 21044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:25.371102 21044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:25.371232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 20976
I20260620 09:20:25.371344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:25.371496 392 external_mini_cluster.cc:1428] Setting key c4febdeb86aa3c93d3562c40a6f40d29
I20260620 09:20:25.371759 21044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebd331723fb4655837031212c7af0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39095 } }
I20260620 09:20:25.371853 21044 leader_election.cc:302] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ebd331723fb4655837031212c7af0c7; no voters:
I20260620 09:20:25.372005 21044 leader_election.cc:288] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:25.372082 21048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:25.372229 21048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [term 1 LEADER]: Becoming Leader. State: Replica: 0ebd331723fb4655837031212c7af0c7, State: Running, Role: LEADER
I20260620 09:20:25.372296 21044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:25.372309 21048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebd331723fb4655837031212c7af0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39095 } }
I20260620 09:20:25.372668 21048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ebd331723fb4655837031212c7af0c7. Latest consensus state: current_term: 1 leader_uuid: "0ebd331723fb4655837031212c7af0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebd331723fb4655837031212c7af0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39095 } } }
I20260620 09:20:25.372723 21050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ebd331723fb4655837031212c7af0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebd331723fb4655837031212c7af0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39095 } } }
I20260620 09:20:25.372788 21048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:25.372879 21050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:25.373093 21053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:25.373627 21053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:25.375742 21053 catalog_manager.cc:1347] Generated new cluster ID: f77f8cb10c1c4426a5520879ec8b51c7
I20260620 09:20:25.375799 21053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:25.377466 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:25.371933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33723 (local address 127.0.0.1:39095)
0620 09:20:25.372103 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:25.372122 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:25.372156 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:20:25.372272 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:25.372279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:25.372338 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:25.372709 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:25.372820 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:25.374224 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:25.375310 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:25.375807 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:25.375912 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:25.376678 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:25.376681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:25.376681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:25.376891 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:25.376924 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:25.376937 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:25.376982 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:25.377220 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 09:20:25.377279 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260620 09:20:25.378136 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:25.382828 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:25.379533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47035 (local address 127.0.0.1:39095)
0620 09:20:25.379565 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:25.379566 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:25.379782 (+ 216us) server_negotiation.cc:407] Connection header received
0620 09:20:25.379857 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:25.379858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:25.379867 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:25.379917 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:25.380039 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:25.380723 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:25.381673 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:25.381950 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:25.382001 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:25.382610 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:25.382611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:25.382612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:25.382616 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:25.382628 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:25.382632 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:25.382676 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:25.382741 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 09:20:25.382762 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:20:25.387702 21053 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:25.388182 21053 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:25.404247 21053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ebd331723fb4655837031212c7af0c7: Generated new TSK 0
I20260620 09:20:25.404433 21053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:25.410559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 20976
2026-06-20T09:20:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:25 dist-test-slave-rq8c krb5kdc[21075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:25 dist-test-slave-rq8c krb5kdc[21075](info): set up 2 sockets
Jun 20 09:20:25 dist-test-slave-rq8c krb5kdc[21075](info): commencing operation
krb5kdc: starting...
W20260620 09:20:26.351329 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:26 dist-test-slave-rq8c krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:20:26.455689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45701
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:26.533025 21088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:26.533229 21088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:26.533252 21088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:26.534348 21088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:26.534382 21088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:26.534394 21088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:26.534405 21088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:26.534416 21088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:26.535836 21088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:26.536023 21088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:26.536197 21088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:26.541150 21094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:26.541144 21093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:26.541237 21088 server_base.cc:1034] running on GCE node
W20260620 09:20:26.541168 21096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:26.541589 21088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:26.541868 21088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:26.543020 21088 hybrid_clock.cc:648] HybridClock initialized: now 1781947226543003 us; error 35 us; skew 500 ppm
Jun 20 09:20:26 dist-test-slave-rq8c krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:26.545315 21088 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:20:26.545725 21088 webserver.cc:460] Webserver started at http://127.0.0.1:38651/ using document root <none> and password file <none>
I20260620 09:20:26.545912 21088 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:26.545957 21088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:26.546085 21088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:26.547093 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "5668e24edd5b48f1b9f4e357a9783594"
format_stamp: "Formatted at 2026-06-20 09:20:26 on dist-test-slave-rq8c"
server_key: "89d5a0e5f4ee6223ae2b237670533a88"
server_key_iv: "49dcab2ffdc0849771056ef01baab628"
server_key_version: "encryptionkey@0"
I20260620 09:20:26.547403 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "5668e24edd5b48f1b9f4e357a9783594"
format_stamp: "Formatted at 2026-06-20 09:20:26 on dist-test-slave-rq8c"
server_key: "89d5a0e5f4ee6223ae2b237670533a88"
server_key_iv: "49dcab2ffdc0849771056ef01baab628"
server_key_version: "encryptionkey@0"
I20260620 09:20:26.548662 21088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:26.549330 21103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:26.549499 21088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:26.549575 21088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "5668e24edd5b48f1b9f4e357a9783594"
format_stamp: "Formatted at 2026-06-20 09:20:26 on dist-test-slave-rq8c"
server_key: "89d5a0e5f4ee6223ae2b237670533a88"
server_key_iv: "49dcab2ffdc0849771056ef01baab628"
server_key_version: "encryptionkey@0"
I20260620 09:20:26.549639 21088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:26.552789 21088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:26.554598 21088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:26.554719 21088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:26.558257 21088 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45701
I20260620 09:20:26.558287 21155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45701 every 8 connection(s)
I20260620 09:20:26.558565 21088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:26.559439 21156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:26.559578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 21088
I20260620 09:20:26.559701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:26.559855 392 external_mini_cluster.cc:1428] Setting key a3ff8acfdec448098401095c5a7910a2
I20260620 09:20:26.561959 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:26.560286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45599 (local address 127.0.0.1:45701)
0620 09:20:26.560433 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:26.560440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:26.560462 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:20:26.560873 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:26.560879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:26.560903 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:26.561017 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:26.561593 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:26.561596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:26.561597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:26.561600 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:26.561633 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:26.561642 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:26.561692 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:26.561762 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 09:20:26.561792 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260620 09:20:26.562557 21156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594: Bootstrap starting.
I20260620 09:20:26.562747 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:26.563115 21156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:26.563352 21156 log.cc:824] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:26.563992 21156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594: No bootstrap required, opened a new log
I20260620 09:20:26.564967 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:26.564053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47223 (local address 127.0.0.1:45701)
0620 09:20:26.564087 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:26.564089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:26.564189 (+ 100us) server_negotiation.cc:407] Connection header received
0620 09:20:26.564240 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:26.564241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:26.564246 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:26.564289 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:26.564786 (+ 497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:26.564788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:26.564788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:26.564792 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:26.564804 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:26.564808 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:26.564851 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:26.564900 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 09:20:26.564912 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:20:26.565871 21156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5668e24edd5b48f1b9f4e357a9783594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45701 } }
I20260620 09:20:26.565999 21156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:26.566028 21156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5668e24edd5b48f1b9f4e357a9783594, State: Initialized, Role: FOLLOWER
I20260620 09:20:26.566133 21156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5668e24edd5b48f1b9f4e357a9783594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45701 } }
I20260620 09:20:26.566200 21156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:26.566229 21156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:26.566273 21156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:26.566766 21156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5668e24edd5b48f1b9f4e357a9783594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45701 } }
I20260620 09:20:26.566854 21156 leader_election.cc:302] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5668e24edd5b48f1b9f4e357a9783594; no voters:
I20260620 09:20:26.567005 21156 leader_election.cc:288] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:26.567034 21166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:26.567175 21166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [term 1 LEADER]: Becoming Leader. State: Replica: 5668e24edd5b48f1b9f4e357a9783594, State: Running, Role: LEADER
I20260620 09:20:26.567215 21156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:26.567289 21166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5668e24edd5b48f1b9f4e357a9783594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45701 } }
I20260620 09:20:26.567646 21167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5668e24edd5b48f1b9f4e357a9783594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5668e24edd5b48f1b9f4e357a9783594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45701 } } }
I20260620 09:20:26.567675 21168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5668e24edd5b48f1b9f4e357a9783594. Latest consensus state: current_term: 1 leader_uuid: "5668e24edd5b48f1b9f4e357a9783594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5668e24edd5b48f1b9f4e357a9783594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45701 } } }
I20260620 09:20:26.567739 21167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:26.567744 21168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:26.567929 21174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:26.568348 21174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:26.569730 21174 catalog_manager.cc:1347] Generated new cluster ID: 1ea42116e7834389b71ec79357d7c16b
I20260620 09:20:26.569770 21174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:26.603305 21174 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:26.603502 21174 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:26.608176 21174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5668e24edd5b48f1b9f4e357a9783594: Generated new TSK 0
I20260620 09:20:26.608340 21174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:26.615711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 21088
2026-06-20T09:20:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:26 dist-test-slave-rq8c krb5kdc[21187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:26 dist-test-slave-rq8c krb5kdc[21187](info): set up 2 sockets
Jun 20 09:20:26 dist-test-slave-rq8c krb5kdc[21187](info): commencing operation
krb5kdc: starting...
W20260620 09:20:27.559365 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:27 dist-test-slave-rq8c krb5kdc[21187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:20:27.662108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:27.738935 21200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:27.739142 21200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:27.739163 21200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:27.740332 21200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:27.740373 21200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:27.740386 21200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:27.740397 21200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:27.740407 21200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:27.741845 21200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:27.742070 21200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:27.742261 21200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:27.746742 21205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:27.746773 21200 server_base.cc:1034] running on GCE node
W20260620 09:20:27.746788 21208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:27.746726 21206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:27.747197 21200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:27.747473 21200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:27.748607 21200 hybrid_clock.cc:648] HybridClock initialized: now 1781947227748584 us; error 35 us; skew 500 ppm
Jun 20 09:20:27 dist-test-slave-rq8c krb5kdc[21187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:27.753734 21200 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:20:27.754153 21200 webserver.cc:460] Webserver started at http://127.0.0.1:44805/ using document root <none> and password file <none>
I20260620 09:20:27.754334 21200 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:27.754372 21200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:27.754468 21200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:27.755430 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "db931cb502ff4a919157afea85b73a02"
format_stamp: "Formatted at 2026-06-20 09:20:27 on dist-test-slave-rq8c"
server_key: "23cac44a9fe546d6c68b25a9221c90ad"
server_key_iv: "b08a99549ddad9c4e039a69a9a4d2fcd"
server_key_version: "encryptionkey@0"
I20260620 09:20:27.755782 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "db931cb502ff4a919157afea85b73a02"
format_stamp: "Formatted at 2026-06-20 09:20:27 on dist-test-slave-rq8c"
server_key: "23cac44a9fe546d6c68b25a9221c90ad"
server_key_iv: "b08a99549ddad9c4e039a69a9a4d2fcd"
server_key_version: "encryptionkey@0"
I20260620 09:20:27.756958 21200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:20:27.757674 21215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:27.757885 21200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:20:27.757978 21200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "db931cb502ff4a919157afea85b73a02"
format_stamp: "Formatted at 2026-06-20 09:20:27 on dist-test-slave-rq8c"
server_key: "23cac44a9fe546d6c68b25a9221c90ad"
server_key_iv: "b08a99549ddad9c4e039a69a9a4d2fcd"
server_key_version: "encryptionkey@0"
I20260620 09:20:27.758054 21200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:27.770855 21200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:27.773027 21200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:27.773142 21200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:27.776788 21200 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43013
I20260620 09:20:27.776832 21267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43013 every 8 connection(s)
I20260620 09:20:27.777122 21200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:27.777907 21268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:27.780174 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02: Bootstrap starting.
I20260620 09:20:27.780683 21268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:27.780900 21268 log.cc:824] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:27.781556 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02: No bootstrap required, opened a new log
I20260620 09:20:27.783073 21268 raft_consensus.cc:348] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db931cb502ff4a919157afea85b73a02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43013 } }
I20260620 09:20:27.783210 21268 raft_consensus.cc:374] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:27.783231 21268 raft_consensus.cc:729] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db931cb502ff4a919157afea85b73a02, State: Initialized, Role: FOLLOWER
I20260620 09:20:27.783327 21268 consensus_queue.cc:260] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db931cb502ff4a919157afea85b73a02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43013 } }
I20260620 09:20:27.783380 21268 raft_consensus.cc:388] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:27.783406 21268 raft_consensus.cc:482] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:27.783443 21268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:27.783972 21268 raft_consensus.cc:504] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db931cb502ff4a919157afea85b73a02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43013 } }
I20260620 09:20:27.784062 21268 leader_election.cc:302] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db931cb502ff4a919157afea85b73a02; no voters:
I20260620 09:20:27.784217 21268 leader_election.cc:288] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:27.784255 21271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:27.784405 21271 raft_consensus.cc:686] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [term 1 LEADER]: Becoming Leader. State: Replica: db931cb502ff4a919157afea85b73a02, State: Running, Role: LEADER
I20260620 09:20:27.784463 21268 sys_catalog.cc:564] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:27.784495 21271 consensus_queue.cc:237] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db931cb502ff4a919157afea85b73a02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43013 } }
I20260620 09:20:27.784842 21272 sys_catalog.cc:455] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db931cb502ff4a919157afea85b73a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db931cb502ff4a919157afea85b73a02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43013 } } }
I20260620 09:20:27.784845 21273 sys_catalog.cc:455] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db931cb502ff4a919157afea85b73a02. Latest consensus state: current_term: 1 leader_uuid: "db931cb502ff4a919157afea85b73a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db931cb502ff4a919157afea85b73a02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43013 } } }
I20260620 09:20:27.784955 21272 sys_catalog.cc:458] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:27.784971 21273 sys_catalog.cc:458] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:27.785152 21275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:27.785640 21275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:27.786281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 21200
I20260620 09:20:27.786393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:27.786568 392 external_mini_cluster.cc:1428] Setting key 09e0ee60b5cf6cfceca10f830836ba87
I20260620 09:20:27.787199 21275 catalog_manager.cc:1347] Generated new cluster ID: 6b6ac93c377541468409e76ff6b7201d
I20260620 09:20:27.787248 21275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 09:20:27 dist-test-slave-rq8c krb5kdc[21187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:20:27.794190 21275 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:27.794823 21275 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:27.797644 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:27.786983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:57831 (local address 127.0.0.1:43013)
0620 09:20:27.787157 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:27.787166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:27.787200 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:20:27.787310 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:27.787317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:27.787373 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:27.787516 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:27.788188 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:27.789713 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:27.790708 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:27.791324 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:27.791454 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:27.794112 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:27.794116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:27.794117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:27.794129 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:27.796069 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:27.796513 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:27.796516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:27.796516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:27.796607 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:27.796802 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:27.796802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:27.796803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:27.797056 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:27.797196 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:27.797315 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 09:20:27.797419 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260620 09:20:27.798334 21275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db931cb502ff4a919157afea85b73a02: Generated new TSK 0
I20260620 09:20:27.798485 21275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:27.798509 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:27.806210 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:27.799715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:49403 (local address 127.0.0.1:43013)
0620 09:20:27.799752 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:27.799754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:27.799828 (+ 74us) server_negotiation.cc:407] Connection header received
0620 09:20:27.799896 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:27.799896 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:27.799903 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:27.799944 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:27.800540 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:27.801027 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:27.802060 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:27.802266 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:27.802329 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:27.803601 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:27.803602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:27.803603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:27.803609 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:27.805256 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:27.805622 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:27.805622 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:27.805623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:27.805663 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:27.805841 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:27.805841 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:27.805841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:27.805921 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:27.805987 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:27.806113 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 09:20:27.806156 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:20:27.807948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 21200
W20260620 09:20:27.811679 21185 connection.cc:535] client connection to 127.0.0.1:43013 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43013): Connection reset by peer (error 104)
2026-06-20T09:20:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1191 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:27 dist-test-slave-rq8c krb5kdc[21299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:27 dist-test-slave-rq8c krb5kdc[21299](info): set up 2 sockets
Jun 20 09:20:27 dist-test-slave-rq8c krb5kdc[21299](info): commencing operation
krb5kdc: starting...
W20260620 09:20:28.747426 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:28 dist-test-slave-rq8c krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:20:28.854332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:28.932873 21312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:28.933086 21312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:28.933110 21312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:28.934284 21312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:28.934329 21312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:28.934342 21312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:28.934353 21312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:28.934363 21312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:28.935897 21312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:28.936132 21312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:28.936344 21312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:28.941017 21318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:28.941078 21312 server_base.cc:1034] running on GCE node
W20260620 09:20:28.941048 21320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:28.941039 21317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:28.941358 21312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:28.941643 21312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:28.942795 21312 hybrid_clock.cc:648] HybridClock initialized: now 1781947228942782 us; error 30 us; skew 500 ppm
Jun 20 09:20:28 dist-test-slave-rq8c krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:28.945228 21312 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:20:28.945641 21312 webserver.cc:460] Webserver started at http://127.0.0.1:36435/ using document root <none> and password file <none>
I20260620 09:20:28.945837 21312 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:28.945883 21312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:28.946004 21312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:28.947024 21312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "e29d5ed9c2d148e1a657fe9374106ce2"
format_stamp: "Formatted at 2026-06-20 09:20:28 on dist-test-slave-rq8c"
server_key: "05ac45833a0cfe61ec1a4f28eb2be7ff"
server_key_iv: "da528b18289e8b4571e6f7d6d4d5b46e"
server_key_version: "encryptionkey@0"
I20260620 09:20:28.947345 21312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "e29d5ed9c2d148e1a657fe9374106ce2"
format_stamp: "Formatted at 2026-06-20 09:20:28 on dist-test-slave-rq8c"
server_key: "05ac45833a0cfe61ec1a4f28eb2be7ff"
server_key_iv: "da528b18289e8b4571e6f7d6d4d5b46e"
server_key_version: "encryptionkey@0"
I20260620 09:20:28.948673 21312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:28.949348 21327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:28.949513 21312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:28.949592 21312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "e29d5ed9c2d148e1a657fe9374106ce2"
format_stamp: "Formatted at 2026-06-20 09:20:28 on dist-test-slave-rq8c"
server_key: "05ac45833a0cfe61ec1a4f28eb2be7ff"
server_key_iv: "da528b18289e8b4571e6f7d6d4d5b46e"
server_key_version: "encryptionkey@0"
I20260620 09:20:28.949651 21312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:28.965854 21312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:28.967686 21312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:28.967823 21312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:28.971510 21312 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46201
I20260620 09:20:28.971545 21379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46201 every 8 connection(s)
I20260620 09:20:28.971873 21312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:28.972697 21380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:28.974947 21380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2: Bootstrap starting.
I20260620 09:20:28.975469 21380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:28.975744 21380 log.cc:824] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:28.976423 21380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2: No bootstrap required, opened a new log
I20260620 09:20:28.977974 21380 raft_consensus.cc:348] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29d5ed9c2d148e1a657fe9374106ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260620 09:20:28.978108 21380 raft_consensus.cc:374] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:28.978128 21380 raft_consensus.cc:729] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e29d5ed9c2d148e1a657fe9374106ce2, State: Initialized, Role: FOLLOWER
I20260620 09:20:28.978264 21380 consensus_queue.cc:260] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29d5ed9c2d148e1a657fe9374106ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260620 09:20:28.978327 21380 raft_consensus.cc:388] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:28.978348 21380 raft_consensus.cc:482] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:28.978394 21380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:28.978688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 21312
I20260620 09:20:28.978775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:28.978935 392 external_mini_cluster.cc:1428] Setting key 2f866fa91026d44bc6306502c101cdd5
I20260620 09:20:28.978932 21380 raft_consensus.cc:504] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29d5ed9c2d148e1a657fe9374106ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260620 09:20:28.979024 21380 leader_election.cc:302] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e29d5ed9c2d148e1a657fe9374106ce2; no voters:
I20260620 09:20:28.979187 21380 leader_election.cc:288] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:28.979234 21383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:28.979445 21380 sys_catalog.cc:564] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:28.979599 21383 raft_consensus.cc:686] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [term 1 LEADER]: Becoming Leader. State: Replica: e29d5ed9c2d148e1a657fe9374106ce2, State: Running, Role: LEADER
I20260620 09:20:28.979764 21383 consensus_queue.cc:237] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29d5ed9c2d148e1a657fe9374106ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260620 09:20:28.980105 21384 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e29d5ed9c2d148e1a657fe9374106ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29d5ed9c2d148e1a657fe9374106ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } } }
I20260620 09:20:28.980237 21384 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:28.980163 21387 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e29d5ed9c2d148e1a657fe9374106ce2. Latest consensus state: current_term: 1 leader_uuid: "e29d5ed9c2d148e1a657fe9374106ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29d5ed9c2d148e1a657fe9374106ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } } }
I20260620 09:20:28.980470 21387 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:28.980928 21394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:28.981416 21394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:28.983343 21394 catalog_manager.cc:1347] Generated new cluster ID: 8ae57f31f1f54bef8e38b9b10a359f97
I20260620 09:20:28.983392 21394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:28.984517 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:28.979440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:55793 (local address 127.0.0.1:46201)
0620 09:20:28.979688 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:28.979697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:28.979732 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:20:28.979843 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:28.979850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:28.979906 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:28.980056 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:28.980322 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:28.981746 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:28.982617 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:28.983074 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:28.983169 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:28.983852 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:28.983855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:28.983856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:28.984065 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:28.984102 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:28.984117 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:28.984205 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:28.984277 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 09:20:28.984304 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260620 09:20:28.985529 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:28.990890 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:28.986987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:42883 (local address 127.0.0.1:46201)
0620 09:20:28.987022 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:28.987024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:28.987129 (+ 105us) server_negotiation.cc:407] Connection header received
0620 09:20:28.987215 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:28.987215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:28.987225 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:28.987252 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:28.987449 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:28.988487 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:28.989501 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:28.989826 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:28.989866 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:28.990621 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:28.990623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:28.990623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:28.990628 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:28.990641 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:28.990645 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:28.990676 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:28.990802 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 09:20:28.990814 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:20:29.008940 21394 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:29.009546 21394 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:29.012787 21394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e29d5ed9c2d148e1a657fe9374106ce2: Generated new TSK 0
I20260620 09:20:29.012966 21394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:29.018196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 21312
W20260620 09:20:29.021776 21297 connection.cc:535] client connection to 127.0.0.1:46201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46201): Connection reset by peer (error 104)
2026-06-20T09:20:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:29 dist-test-slave-rq8c krb5kdc[21411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:29 dist-test-slave-rq8c krb5kdc[21411](info): set up 2 sockets
Jun 20 09:20:29 dist-test-slave-rq8c krb5kdc[21411](info): commencing operation
krb5kdc: starting...
W20260620 09:20:29.959385 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:30 dist-test-slave-rq8c krb5kdc[21411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:20:30.066018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:30.144819 21424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:30.145031 21424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:30.145063 21424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:30.146214 21424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:30.146256 21424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:30.146275 21424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:30.146291 21424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:30.146307 21424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:30.147825 21424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:30.148057 21424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:30.148265 21424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:30.152704 21429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:30.152746 21430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:30.152705 21432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:30.152721 21424 server_base.cc:1034] running on GCE node
I20260620 09:20:30.153136 21424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:30.153410 21424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:30.154551 21424 hybrid_clock.cc:648] HybridClock initialized: now 1781947230154538 us; error 29 us; skew 500 ppm
Jun 20 09:20:30 dist-test-slave-rq8c krb5kdc[21411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:30.156958 21424 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:20:30.157349 21424 webserver.cc:460] Webserver started at http://127.0.0.1:44669/ using document root <none> and password file <none>
I20260620 09:20:30.157536 21424 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:30.157577 21424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:30.157706 21424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:30.158718 21424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "332703888e2b47218d04fd0359dce78b"
format_stamp: "Formatted at 2026-06-20 09:20:30 on dist-test-slave-rq8c"
server_key: "26502bcf0d1b9242d008142d41ba105b"
server_key_iv: "1812ef307feb3c432ae6997b01d9a96d"
server_key_version: "encryptionkey@0"
I20260620 09:20:30.159039 21424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "332703888e2b47218d04fd0359dce78b"
format_stamp: "Formatted at 2026-06-20 09:20:30 on dist-test-slave-rq8c"
server_key: "26502bcf0d1b9242d008142d41ba105b"
server_key_iv: "1812ef307feb3c432ae6997b01d9a96d"
server_key_version: "encryptionkey@0"
I20260620 09:20:30.160403 21424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:30.161058 21439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:30.161218 21424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:30.161299 21424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "332703888e2b47218d04fd0359dce78b"
format_stamp: "Formatted at 2026-06-20 09:20:30 on dist-test-slave-rq8c"
server_key: "26502bcf0d1b9242d008142d41ba105b"
server_key_iv: "1812ef307feb3c432ae6997b01d9a96d"
server_key_version: "encryptionkey@0"
I20260620 09:20:30.161358 21424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:30.164494 21424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:30.166229 21424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:30.166360 21424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:30.169945 21424 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40643
I20260620 09:20:30.169976 21491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40643 every 8 connection(s)
I20260620 09:20:30.170288 21424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:30.171136 21492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:30.173348 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b: Bootstrap starting.
I20260620 09:20:30.173866 21492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:30.174120 21492 log.cc:824] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:30.174731 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b: No bootstrap required, opened a new log
I20260620 09:20:30.176291 21492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332703888e2b47218d04fd0359dce78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40643 } }
I20260620 09:20:30.176419 21492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:30.176438 21492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332703888e2b47218d04fd0359dce78b, State: Initialized, Role: FOLLOWER
I20260620 09:20:30.176574 21492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332703888e2b47218d04fd0359dce78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40643 } }
I20260620 09:20:30.176630 21492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:30.176649 21492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:30.176690 21492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:30.177172 21492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332703888e2b47218d04fd0359dce78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40643 } }
I20260620 09:20:30.177253 21492 leader_election.cc:302] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 332703888e2b47218d04fd0359dce78b; no voters:
I20260620 09:20:30.177392 21492 leader_election.cc:288] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:30.177431 21495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:30.177572 21495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [term 1 LEADER]: Becoming Leader. State: Replica: 332703888e2b47218d04fd0359dce78b, State: Running, Role: LEADER
I20260620 09:20:30.177624 21492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:30.177656 21495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332703888e2b47218d04fd0359dce78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40643 } }
I20260620 09:20:30.177990 21497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 332703888e2b47218d04fd0359dce78b. Latest consensus state: current_term: 1 leader_uuid: "332703888e2b47218d04fd0359dce78b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332703888e2b47218d04fd0359dce78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40643 } } }
I20260620 09:20:30.178068 21497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:30.178146 21496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "332703888e2b47218d04fd0359dce78b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332703888e2b47218d04fd0359dce78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40643 } } }
I20260620 09:20:30.178228 21496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:30.178303 21502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:30.178803 21502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:30.180204 21502 catalog_manager.cc:1347] Generated new cluster ID: bdc7c63dc3af42fa80ddec9041c85df1
I20260620 09:20:30.180249 21502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:30.180455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 21424
I20260620 09:20:30.180569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:30.180739 392 external_mini_cluster.cc:1428] Setting key 0c7a01e52731b868fa223e076b903a71
I20260620 09:20:30.182843 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:30.181193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:58463 (local address 127.0.0.1:40643)
0620 09:20:30.181367 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:30.181375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:30.181560 (+ 185us) server_negotiation.cc:407] Connection header received
0620 09:20:30.181692 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:30.181699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:30.181728 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:30.181856 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:30.182445 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:30.182449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:30.182449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:30.182452 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:30.182492 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:30.182505 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:30.182569 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:30.182643 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 09:20:30.182681 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260620 09:20:30.183671 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:30.186359 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:30.185195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:59207 (local address 127.0.0.1:40643)
0620 09:20:30.185244 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:30.185246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:30.185478 (+ 232us) server_negotiation.cc:407] Connection header received
0620 09:20:30.185541 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:30.185542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:30.185546 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:30.185594 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:30.186156 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:30.186158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:30.186158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:30.186162 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:30.186175 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:30.186186 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:30.186231 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:30.186292 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 09:20:30.186303 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 09:20:30.193419 21502 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:30.193575 21502 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:30.200166 21502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 332703888e2b47218d04fd0359dce78b: Generated new TSK 0
I20260620 09:20:30.200377 21502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:30.211730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 21424
2026-06-20T09:20:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1193 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:30 dist-test-slave-rq8c krb5kdc[21523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:30 dist-test-slave-rq8c krb5kdc[21523](info): set up 2 sockets
Jun 20 09:20:30 dist-test-slave-rq8c krb5kdc[21523](info): commencing operation
krb5kdc: starting...
W20260620 09:20:31.167434 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:31 dist-test-slave-rq8c krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:20:31.278124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:31.356359 21536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:31.356552 21536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:31.356573 21536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:31.357705 21536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:31.357746 21536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:31.357758 21536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:31.357769 21536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:31.357779 21536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:31.359200 21536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:31.359424 21536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:31.359663 21536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:31.364207 21544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:31.364214 21542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:31.364338 21536 server_base.cc:1034] running on GCE node
W20260620 09:20:31.364223 21541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:31.364632 21536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:31.364874 21536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:31.366024 21536 hybrid_clock.cc:648] HybridClock initialized: now 1781947231366003 us; error 38 us; skew 500 ppm
Jun 20 09:20:31 dist-test-slave-rq8c krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:31.368393 21536 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:20:31.368769 21536 webserver.cc:460] Webserver started at http://127.0.0.1:38187/ using document root <none> and password file <none>
I20260620 09:20:31.368980 21536 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:31.369032 21536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:31.369163 21536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:31.370177 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "d9179d75f8a441b7b0ea31e84959f420"
format_stamp: "Formatted at 2026-06-20 09:20:31 on dist-test-slave-rq8c"
server_key: "b2d5c096ec7e53ec743088b8c2ba02cb"
server_key_iv: "dc945f3e26c5d5a9eda919a3c03e59dd"
server_key_version: "encryptionkey@0"
I20260620 09:20:31.370483 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9179d75f8a441b7b0ea31e84959f420"
format_stamp: "Formatted at 2026-06-20 09:20:31 on dist-test-slave-rq8c"
server_key: "b2d5c096ec7e53ec743088b8c2ba02cb"
server_key_iv: "dc945f3e26c5d5a9eda919a3c03e59dd"
server_key_version: "encryptionkey@0"
I20260620 09:20:31.371727 21536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:20:31.372393 21551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:31.372536 21536 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:31.372612 21536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "d9179d75f8a441b7b0ea31e84959f420"
format_stamp: "Formatted at 2026-06-20 09:20:31 on dist-test-slave-rq8c"
server_key: "b2d5c096ec7e53ec743088b8c2ba02cb"
server_key_iv: "dc945f3e26c5d5a9eda919a3c03e59dd"
server_key_version: "encryptionkey@0"
I20260620 09:20:31.372680 21536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:31.383518 21536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:31.385603 21536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:31.385733 21536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:31.389390 21536 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32977
I20260620 09:20:31.389459 21603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32977 every 8 connection(s)
I20260620 09:20:31.389740 21536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:31.390556 21604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:31.392520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 21536
I20260620 09:20:31.392630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:31.392783 392 external_mini_cluster.cc:1428] Setting key 98ffeabcc65479c65e1aa292e89028e1
I20260620 09:20:31.392966 21604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420: Bootstrap starting.
I20260620 09:20:31.393601 21604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:31.393862 21604 log.cc:824] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:31.394502 21604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420: No bootstrap required, opened a new log
I20260620 09:20:31.396719 21604 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9179d75f8a441b7b0ea31e84959f420" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } }
I20260620 09:20:31.396883 21604 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:31.396914 21604 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9179d75f8a441b7b0ea31e84959f420, State: Initialized, Role: FOLLOWER
I20260620 09:20:31.397033 21604 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9179d75f8a441b7b0ea31e84959f420" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } }
I20260620 09:20:31.397101 21604 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:31.397136 21604 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:31.397178 21604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:31.397831 21604 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9179d75f8a441b7b0ea31e84959f420" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } }
I20260620 09:20:31.397936 21604 leader_election.cc:302] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9179d75f8a441b7b0ea31e84959f420; no voters:
I20260620 09:20:31.398092 21604 leader_election.cc:288] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:31.398135 21609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:31.398283 21609 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [term 1 LEADER]: Becoming Leader. State: Replica: d9179d75f8a441b7b0ea31e84959f420, State: Running, Role: LEADER
I20260620 09:20:31.398339 21604 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:31.398380 21609 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9179d75f8a441b7b0ea31e84959f420" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } }
I20260620 09:20:31.398682 21609 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9179d75f8a441b7b0ea31e84959f420. Latest consensus state: current_term: 1 leader_uuid: "d9179d75f8a441b7b0ea31e84959f420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9179d75f8a441b7b0ea31e84959f420" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } } }
I20260620 09:20:31.398758 21609 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:31.398690 21611 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9179d75f8a441b7b0ea31e84959f420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9179d75f8a441b7b0ea31e84959f420" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } } }
I20260620 09:20:31.398908 21611 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420 [sys.catalog]: This master's current role is: LEADER
Jun 20 09:20:31 dist-test-slave-rq8c krb5kdc[21523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260620 09:20:31.399907 21624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:20:31.399960 21624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:20:31.400611 21616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:31.400770 21616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:31.402065 21616 catalog_manager.cc:1347] Generated new cluster ID: 941eaade9811496797df19d8722316db
I20260620 09:20:31.402112 21616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:31.403709 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:31.393297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:45477 (local address 127.0.0.1:32977)
0620 09:20:31.393455 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:31.393464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:31.393490 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:20:31.393922 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:31.393931 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:31.393992 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:31.394126 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:31.394697 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:31.396043 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:31.396770 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:31.397362 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:31.397469 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:31.400080 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:31.400084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:31.400084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:31.400095 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:31.402104 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:31.402537 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:31.402540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:31.402540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:31.402637 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:31.402862 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:31.402863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:31.402863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:31.403127 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:31.403265 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:31.403388 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 09:20:31.403482 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260620 09:20:31.404598 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:31.414283 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:31.405955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:52443 (local address 127.0.0.1:32977)
0620 09:20:31.405990 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:31.405992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:31.407764 (+ 1772us) server_negotiation.cc:407] Connection header received
0620 09:20:31.407826 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:31.407827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:31.407835 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:31.407889 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:31.408482 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:31.409201 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:31.410142 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:31.410444 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:31.410498 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:31.411793 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:31.411794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:31.411795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:31.411801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:31.413197 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:31.413580 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:31.413581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:31.413581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:31.413628 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:31.413808 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:31.413808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:31.413809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:31.413917 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:31.414002 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:31.414140 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 09:20:31.414195 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 09:20:31.416965 21616 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:31.417676 21616 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:31.426647 21616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9179d75f8a441b7b0ea31e84959f420: Generated new TSK 0
I20260620 09:20:31.426856 21616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:31.433768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 21536
W20260620 09:20:31.437498 21521 connection.cc:535] client connection to 127.0.0.1:32977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32977): Connection reset by peer (error 104)
2026-06-20T09:20:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:31 dist-test-slave-rq8c krb5kdc[21635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:31 dist-test-slave-rq8c krb5kdc[21635](info): set up 2 sockets
Jun 20 09:20:31 dist-test-slave-rq8c krb5kdc[21635](info): commencing operation
krb5kdc: starting...
W20260620 09:20:32.379400 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:32 dist-test-slave-rq8c krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:20:32.486095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:32.564036 21648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:32.564249 21648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:32.564271 21648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:32.565382 21648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:32.565418 21648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:32.565430 21648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:32.565442 21648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:32.565452 21648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:32.566856 21648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:32.567049 21648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:32.567219 21648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:32.571743 21656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:32.571743 21654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:32.571771 21653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:32.571817 21648 server_base.cc:1034] running on GCE node
I20260620 09:20:32.572178 21648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:32.572456 21648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:32.573613 21648 hybrid_clock.cc:648] HybridClock initialized: now 1781947232573594 us; error 29 us; skew 500 ppm
Jun 20 09:20:32 dist-test-slave-rq8c krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:32.575889 21648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:20:32.576385 21648 webserver.cc:460] Webserver started at http://127.0.0.1:40697/ using document root <none> and password file <none>
I20260620 09:20:32.576589 21648 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:32.576638 21648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:32.576756 21648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:32.577888 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "38a038daaaef4dd59832dfffd9dc1313"
format_stamp: "Formatted at 2026-06-20 09:20:32 on dist-test-slave-rq8c"
server_key: "aa74d66a3635306aec9367a95e35188f"
server_key_iv: "8c00a71af8b729d799444f5d216def72"
server_key_version: "encryptionkey@0"
I20260620 09:20:32.578271 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "38a038daaaef4dd59832dfffd9dc1313"
format_stamp: "Formatted at 2026-06-20 09:20:32 on dist-test-slave-rq8c"
server_key: "aa74d66a3635306aec9367a95e35188f"
server_key_iv: "8c00a71af8b729d799444f5d216def72"
server_key_version: "encryptionkey@0"
I20260620 09:20:32.579537 21648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:32.580240 21663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:32.580379 21648 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:32.580461 21648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "38a038daaaef4dd59832dfffd9dc1313"
format_stamp: "Formatted at 2026-06-20 09:20:32 on dist-test-slave-rq8c"
server_key: "aa74d66a3635306aec9367a95e35188f"
server_key_iv: "8c00a71af8b729d799444f5d216def72"
server_key_version: "encryptionkey@0"
I20260620 09:20:32.580528 21648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:32.595611 21648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:32.597554 21648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:32.597689 21648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:32.601502 21648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34307
I20260620 09:20:32.601529 21715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34307 every 8 connection(s)
I20260620 09:20:32.601845 21648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:32.602731 21716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:32.604931 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313: Bootstrap starting.
I20260620 09:20:32.605459 21716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:32.605676 21716 log.cc:824] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:32.606284 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313: No bootstrap required, opened a new log
I20260620 09:20:32.607808 21716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a038daaaef4dd59832dfffd9dc1313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34307 } }
I20260620 09:20:32.607941 21716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:32.607962 21716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a038daaaef4dd59832dfffd9dc1313, State: Initialized, Role: FOLLOWER
I20260620 09:20:32.608072 21716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a038daaaef4dd59832dfffd9dc1313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34307 } }
I20260620 09:20:32.608151 21716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:32.608178 21716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:32.608209 21716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:32.608654 21716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a038daaaef4dd59832dfffd9dc1313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34307 } }
I20260620 09:20:32.608733 21716 leader_election.cc:302] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38a038daaaef4dd59832dfffd9dc1313; no voters:
I20260620 09:20:32.608851 21716 leader_election.cc:288] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:32.608897 21719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:32.609054 21719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [term 1 LEADER]: Becoming Leader. State: Replica: 38a038daaaef4dd59832dfffd9dc1313, State: Running, Role: LEADER
I20260620 09:20:32.609079 21716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:32.609144 21719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a038daaaef4dd59832dfffd9dc1313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34307 } }
I20260620 09:20:32.609634 21721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a038daaaef4dd59832dfffd9dc1313. Latest consensus state: current_term: 1 leader_uuid: "38a038daaaef4dd59832dfffd9dc1313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a038daaaef4dd59832dfffd9dc1313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34307 } } }
I20260620 09:20:32.609714 21721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:32.609897 21720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38a038daaaef4dd59832dfffd9dc1313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a038daaaef4dd59832dfffd9dc1313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34307 } } }
I20260620 09:20:32.610008 21720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:32.610117 21728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:32.610347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 21648
I20260620 09:20:32.610443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:32.610586 21728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:32.610636 392 external_mini_cluster.cc:1428] Setting key 805efc401c1f1a40c6b94d83741f32a5
I20260620 09:20:32.612099 21728 catalog_manager.cc:1347] Generated new cluster ID: 42fa41bd7cdd46cf94d8460c51d2fc06
I20260620 09:20:32.612150 21728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:32.615583 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:32.611046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:37503 (local address 127.0.0.1:34307)
0620 09:20:32.611547 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:32.611556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:32.611584 (+ 28us) server_negotiation.cc:407] Connection header received
0620 09:20:32.611718 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:32.611725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:32.611783 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:32.611920 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:32.612172 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:32.613063 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:32.613816 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:32.614243 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:32.614333 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:32.614983 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:32.614988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:32.614988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:32.615156 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:32.615186 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:32.615197 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:32.615265 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:32.615353 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 09:20:32.615382 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":97,"threads_started":1}
I20260620 09:20:32.616407 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:32.620239 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:32.617730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:52209 (local address 127.0.0.1:34307)
0620 09:20:32.617764 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:32.617766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:32.617773 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:20:32.617791 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:32.617792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:32.617801 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:32.617852 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:32.617977 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:32.618443 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:32.619090 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:32.619327 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:32.619380 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:32.620004 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:32.620006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:32.620006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:32.620010 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:32.620024 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:32.620029 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:32.620079 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:32.620152 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 09:20:32.620165 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:20:32.629522 21728 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:32.630017 21728 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:32.635087 21728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38a038daaaef4dd59832dfffd9dc1313: Generated new TSK 0
I20260620 09:20:32.635258 21728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:32.638876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 21648
W20260620 09:20:32.642540 21633 connection.cc:535] client connection to 127.0.0.1:34307 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34307): Connection reset by peer (error 104)
2026-06-20T09:20:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:32 dist-test-slave-rq8c krb5kdc[21747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:32 dist-test-slave-rq8c krb5kdc[21747](info): set up 2 sockets
Jun 20 09:20:32 dist-test-slave-rq8c krb5kdc[21747](info): commencing operation
krb5kdc: starting...
W20260620 09:20:33.591450 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:33 dist-test-slave-rq8c krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:20:33.698303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:33.775750 21760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:33.776042 21760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:33.776093 21760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:33.777577 21760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:33.777621 21760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:33.777633 21760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:33.777644 21760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:33.777661 21760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:33.779134 21760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:33.779320 21760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:33.779507 21760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:33.783977 21766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:33.784011 21765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:33.784020 21768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:33.784224 21760 server_base.cc:1034] running on GCE node
I20260620 09:20:33.784435 21760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:33.784674 21760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:33.785822 21760 hybrid_clock.cc:648] HybridClock initialized: now 1781947233785795 us; error 42 us; skew 500 ppm
Jun 20 09:20:33 dist-test-slave-rq8c krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:33.788206 21760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:20:33.788636 21760 webserver.cc:460] Webserver started at http://127.0.0.1:39291/ using document root <none> and password file <none>
I20260620 09:20:33.788843 21760 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:33.788889 21760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:33.789006 21760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:33.789983 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "b7f686a1ae7a4fa1acdcb978b3615fe4"
format_stamp: "Formatted at 2026-06-20 09:20:33 on dist-test-slave-rq8c"
server_key: "4313f394c771b0b74badec2dcb1fe0aa"
server_key_iv: "9161c0986c28cc7f0a4d75e9872e479a"
server_key_version: "encryptionkey@0"
I20260620 09:20:33.790287 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "b7f686a1ae7a4fa1acdcb978b3615fe4"
format_stamp: "Formatted at 2026-06-20 09:20:33 on dist-test-slave-rq8c"
server_key: "4313f394c771b0b74badec2dcb1fe0aa"
server_key_iv: "9161c0986c28cc7f0a4d75e9872e479a"
server_key_version: "encryptionkey@0"
I20260620 09:20:33.791427 21760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:20:33.792148 21775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:33.792317 21760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:33.792387 21760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "b7f686a1ae7a4fa1acdcb978b3615fe4"
format_stamp: "Formatted at 2026-06-20 09:20:33 on dist-test-slave-rq8c"
server_key: "4313f394c771b0b74badec2dcb1fe0aa"
server_key_iv: "9161c0986c28cc7f0a4d75e9872e479a"
server_key_version: "encryptionkey@0"
I20260620 09:20:33.792457 21760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:33.795483 21760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:33.797147 21760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:33.797266 21760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:33.800944 21760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45079
I20260620 09:20:33.800973 21827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45079 every 8 connection(s)
I20260620 09:20:33.801262 21760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:33.802125 21828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:33.802343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 21760
I20260620 09:20:33.802451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:33.802614 392 external_mini_cluster.cc:1428] Setting key 6939d9beed5b9a9d6187c607e135ca80
I20260620 09:20:33.804924 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:33.803064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:51413 (local address 127.0.0.1:45079)
0620 09:20:33.803236 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:33.803245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:33.803275 (+ 30us) server_negotiation.cc:407] Connection header received
0620 09:20:33.803803 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:33.803811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:33.803844 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:33.803981 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:33.804544 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:33.804548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:33.804548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:33.804552 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:33.804594 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:33.804607 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:33.804685 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:33.804716 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 09:20:33.804758 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260620 09:20:33.805117 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4: Bootstrap starting.
I20260620 09:20:33.805738 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:33.805783 21828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:33.806041 21828 log.cc:824] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:33.806631 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4: No bootstrap required, opened a new log
I20260620 09:20:33.808063 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:33.807035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.16:53683 (local address 127.0.0.1:45079)
0620 09:20:33.807067 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:33.807069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:33.807259 (+ 190us) server_negotiation.cc:407] Connection header received
0620 09:20:33.807317 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:33.807318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:33.807322 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:33.807367 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:33.807888 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:33.807889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:33.807889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:33.807893 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:33.807905 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:33.807909 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:33.807953 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:33.807993 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 09:20:33.808006 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:20:33.808650 21828 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f686a1ae7a4fa1acdcb978b3615fe4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45079 } }
I20260620 09:20:33.808775 21828 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:33.808795 21828 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f686a1ae7a4fa1acdcb978b3615fe4, State: Initialized, Role: FOLLOWER
I20260620 09:20:33.808883 21828 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f686a1ae7a4fa1acdcb978b3615fe4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45079 } }
I20260620 09:20:33.808961 21828 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:33.808987 21828 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:33.809018 21828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:33.809477 21828 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f686a1ae7a4fa1acdcb978b3615fe4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45079 } }
I20260620 09:20:33.809559 21828 leader_election.cc:302] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7f686a1ae7a4fa1acdcb978b3615fe4; no voters:
I20260620 09:20:33.809691 21828 leader_election.cc:288] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:33.809731 21838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:33.809860 21838 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [term 1 LEADER]: Becoming Leader. State: Replica: b7f686a1ae7a4fa1acdcb978b3615fe4, State: Running, Role: LEADER
I20260620 09:20:33.809911 21828 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:33.809949 21838 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f686a1ae7a4fa1acdcb978b3615fe4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45079 } }
I20260620 09:20:33.810289 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7f686a1ae7a4fa1acdcb978b3615fe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f686a1ae7a4fa1acdcb978b3615fe4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45079 } } }
I20260620 09:20:33.810421 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:33.810290 21840 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7f686a1ae7a4fa1acdcb978b3615fe4. Latest consensus state: current_term: 1 leader_uuid: "b7f686a1ae7a4fa1acdcb978b3615fe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f686a1ae7a4fa1acdcb978b3615fe4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45079 } } }
I20260620 09:20:33.810638 21840 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:33.811020 21847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:33.811460 21847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:33.812857 21847 catalog_manager.cc:1347] Generated new cluster ID: 6cb30f566c4a4057b3af85f8315134ad
I20260620 09:20:33.812906 21847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:33.828074 21847 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:33.828230 21847 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:33.832350 21847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7f686a1ae7a4fa1acdcb978b3615fe4: Generated new TSK 0
I20260620 09:20:33.832499 21847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:33.839397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 21760
2026-06-20T09:20:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1200 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:33 dist-test-slave-rq8c krb5kdc[21859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:33 dist-test-slave-rq8c krb5kdc[21859](info): set up 2 sockets
Jun 20 09:20:33 dist-test-slave-rq8c krb5kdc[21859](info): commencing operation
krb5kdc: starting...
W20260620 09:20:34.787480 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:34 dist-test-slave-rq8c krb5kdc[21859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:20:34.896344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37467
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:34.975528 21872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:34.975759 21872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:34.975781 21872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:34.976925 21872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:34.976959 21872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:34.976972 21872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:34.976984 21872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:34.976994 21872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:34.978404 21872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:34.978590 21872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:34.978760 21872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:34.983564 21877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:34.983676 21880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:34.983654 21878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:34.983819 21872 server_base.cc:1034] running on GCE node
I20260620 09:20:34.984047 21872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:34.984306 21872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:34.985455 21872 hybrid_clock.cc:648] HybridClock initialized: now 1781947234985435 us; error 31 us; skew 500 ppm
Jun 20 09:20:34 dist-test-slave-rq8c krb5kdc[21859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:34.987691 21872 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:20:34.988094 21872 webserver.cc:460] Webserver started at http://127.0.98.62:35007/ using document root <none> and password file <none>
I20260620 09:20:34.988294 21872 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:34.988339 21872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:34.988463 21872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:34.989470 21872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "77c2745bdab249d197207f16e90e73d6"
format_stamp: "Formatted at 2026-06-20 09:20:34 on dist-test-slave-rq8c"
server_key: "e73a5bb9c58fceeb39d4435d1ff7ba63"
server_key_iv: "104a8bbb25fc5f6a6f8bc0facd81e028"
server_key_version: "encryptionkey@0"
I20260620 09:20:34.989782 21872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "77c2745bdab249d197207f16e90e73d6"
format_stamp: "Formatted at 2026-06-20 09:20:34 on dist-test-slave-rq8c"
server_key: "e73a5bb9c58fceeb39d4435d1ff7ba63"
server_key_iv: "104a8bbb25fc5f6a6f8bc0facd81e028"
server_key_version: "encryptionkey@0"
I20260620 09:20:34.991057 21872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 09:20:34.991711 21887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:34.991838 21872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:34.991925 21872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "77c2745bdab249d197207f16e90e73d6"
format_stamp: "Formatted at 2026-06-20 09:20:34 on dist-test-slave-rq8c"
server_key: "e73a5bb9c58fceeb39d4435d1ff7ba63"
server_key_iv: "104a8bbb25fc5f6a6f8bc0facd81e028"
server_key_version: "encryptionkey@0"
I20260620 09:20:34.991981 21872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:35.000375 21872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:35.002286 21872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:35.002413 21872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:35.006029 21872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37467
I20260620 09:20:35.006072 21939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37467 every 8 connection(s)
I20260620 09:20:35.006373 21872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:35.007190 21940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:35.009382 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6: Bootstrap starting.
I20260620 09:20:35.009903 21940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:35.010136 21940 log.cc:824] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:35.010618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 21872
I20260620 09:20:35.010710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:35.010804 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6: No bootstrap required, opened a new log
I20260620 09:20:35.010887 392 external_mini_cluster.cc:1428] Setting key cd107193efa5e4c113fe697735dd9049
I20260620 09:20:35.012641 21940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c2745bdab249d197207f16e90e73d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37467 } }
I20260620 09:20:35.012769 21940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:35.012800 21940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c2745bdab249d197207f16e90e73d6, State: Initialized, Role: FOLLOWER
I20260620 09:20:35.012920 21940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c2745bdab249d197207f16e90e73d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37467 } }
I20260620 09:20:35.012979 21940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:35.013006 21940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:35.013051 21940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:35.013558 21940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c2745bdab249d197207f16e90e73d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37467 } }
I20260620 09:20:35.013645 21940 leader_election.cc:302] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c2745bdab249d197207f16e90e73d6; no voters:
I20260620 09:20:35.013800 21940 leader_election.cc:288] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:35.013842 21945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:35.013962 21945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [term 1 LEADER]: Becoming Leader. State: Replica: 77c2745bdab249d197207f16e90e73d6, State: Running, Role: LEADER
I20260620 09:20:35.014045 21940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:35.014088 21945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c2745bdab249d197207f16e90e73d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37467 } }
I20260620 09:20:35.014573 21946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c2745bdab249d197207f16e90e73d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c2745bdab249d197207f16e90e73d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37467 } } }
I20260620 09:20:35.014634 21946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:35.014439 21945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c2745bdab249d197207f16e90e73d6. Latest consensus state: current_term: 1 leader_uuid: "77c2745bdab249d197207f16e90e73d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c2745bdab249d197207f16e90e73d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37467 } } }
I20260620 09:20:35.014813 21950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:35.015671 21945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:35.016111 21950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 09:20:35 dist-test-slave-rq8c krb5kdc[21859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:35.017972 21950 catalog_manager.cc:1347] Generated new cluster ID: 465aac9244b24216ae85dbed6f802760
I20260620 09:20:35.018018 21950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:35.021987 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:35.011355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34187 (local address 127.0.98.62:37467)
0620 09:20:35.011536 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:35.011543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:35.011566 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:20:35.012025 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:35.012032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:35.012076 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:35.012192 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:35.012760 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:35.014113 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:35.015154 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:35.015820 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:35.015934 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:35.018363 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:35.018367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:35.018367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:35.018378 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:35.020302 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:35.020712 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:35.020714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:35.020715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:35.020817 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:35.021018 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:35.021019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:35.021019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:35.021314 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:35.021491 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:35.021569 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 09:20:35.021666 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260620 09:20:35.022902 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:35.025185 21950 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:35.025643 21950 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:35.029050 21950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77c2745bdab249d197207f16e90e73d6: Generated new TSK 0
I20260620 09:20:35.029227 21950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:35.031198 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:35.024371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43227 (local address 127.0.98.62:37467)
0620 09:20:35.024407 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:35.024409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:35.024485 (+ 76us) server_negotiation.cc:407] Connection header received
0620 09:20:35.024577 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:35.024578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:35.024588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:35.024615 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:35.025280 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:35.026005 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:35.026990 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:35.027292 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:35.027327 (+ 35us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:35.028758 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:35.028759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:35.028760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:35.028766 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:35.030208 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:35.030523 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:35.030523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:35.030524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:35.030577 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:35.030760 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:35.030761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:35.030762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:35.030879 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:35.030980 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:35.031052 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 09:20:35.031108 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:20:35.032868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 21872
W20260620 09:20:35.036443 21857 connection.cc:535] client connection to 127.0.98.62:37467 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37467): Connection reset by peer (error 104)
2026-06-20T09:20:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1193 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:35 dist-test-slave-rq8c krb5kdc[21971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:35 dist-test-slave-rq8c krb5kdc[21971](info): set up 2 sockets
Jun 20 09:20:35 dist-test-slave-rq8c krb5kdc[21971](info): commencing operation
krb5kdc: starting...
W20260620 09:20:35.975449 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:36 dist-test-slave-rq8c krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:20:36.085718 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38465
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:36.165655 21984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:36.165872 21984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:36.165896 21984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:36.167037 21984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:36.167088 21984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:36.167102 21984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:36.167115 21984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:36.167128 21984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:36.168678 21984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:36.168917 21984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:36.169126 21984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:36.173789 21990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:36.173780 21992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:36.173800 21989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:36.173806 21984 server_base.cc:1034] running on GCE node
I20260620 09:20:36.174248 21984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:36.174525 21984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:36.175701 21984 hybrid_clock.cc:648] HybridClock initialized: now 1781947236175682 us; error 38 us; skew 500 ppm
Jun 20 09:20:36 dist-test-slave-rq8c krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:36.178002 21984 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:20:36.178442 21984 webserver.cc:460] Webserver started at http://127.0.98.62:38901/ using document root <none> and password file <none>
I20260620 09:20:36.178647 21984 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:36.178695 21984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:36.178810 21984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:36.179878 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "42271565e20d4de9a89500e1382da70d"
format_stamp: "Formatted at 2026-06-20 09:20:36 on dist-test-slave-rq8c"
server_key: "e58ba786bec83c775318972eebfed3b5"
server_key_iv: "404ec8b100fa33a4a958b2cab359df3d"
server_key_version: "encryptionkey@0"
I20260620 09:20:36.180197 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "42271565e20d4de9a89500e1382da70d"
format_stamp: "Formatted at 2026-06-20 09:20:36 on dist-test-slave-rq8c"
server_key: "e58ba786bec83c775318972eebfed3b5"
server_key_iv: "404ec8b100fa33a4a958b2cab359df3d"
server_key_version: "encryptionkey@0"
I20260620 09:20:36.181459 21984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:20:36.182140 21999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:36.182317 21984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:36.182399 21984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "42271565e20d4de9a89500e1382da70d"
format_stamp: "Formatted at 2026-06-20 09:20:36 on dist-test-slave-rq8c"
server_key: "e58ba786bec83c775318972eebfed3b5"
server_key_iv: "404ec8b100fa33a4a958b2cab359df3d"
server_key_version: "encryptionkey@0"
I20260620 09:20:36.182463 21984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:36.193368 21984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:36.195238 21984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:36.195376 21984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:36.199085 21984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38465
I20260620 09:20:36.199128 22051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38465 every 8 connection(s)
I20260620 09:20:36.199451 21984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:36.200035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 21984
I20260620 09:20:36.200137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:36.200294 392 external_mini_cluster.cc:1428] Setting key cfa18dac94e2165d7932bd04c1d4f99f
I20260620 09:20:36.200300 22052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:36.203307 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d: Bootstrap starting.
I20260620 09:20:36.203949 22052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:36.204201 22052 log.cc:824] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:36.204866 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d: No bootstrap required, opened a new log
I20260620 09:20:36.205562 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:36.200780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42845 (local address 127.0.98.62:38465)
0620 09:20:36.200945 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:36.200954 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:36.200980 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:20:36.201405 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:36.201411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:36.201458 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:36.201568 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:36.201776 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:36.202627 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:36.203696 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:36.204135 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:36.204224 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:36.204897 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:36.204901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:36.204901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:36.205101 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:36.205138 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:36.205153 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:36.205233 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:36.205315 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 09:20:36.205344 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260620 09:20:36.206523 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:36.206943 22052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42271565e20d4de9a89500e1382da70d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38465 } }
I20260620 09:20:36.207077 22052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:36.207098 22052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42271565e20d4de9a89500e1382da70d, State: Initialized, Role: FOLLOWER
I20260620 09:20:36.207214 22052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42271565e20d4de9a89500e1382da70d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38465 } }
I20260620 09:20:36.207278 22052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:36.207299 22052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:36.207343 22052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:36.208092 22052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42271565e20d4de9a89500e1382da70d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38465 } }
I20260620 09:20:36.208205 22052 leader_election.cc:302] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42271565e20d4de9a89500e1382da70d; no voters:
I20260620 09:20:36.208379 22052 leader_election.cc:288] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:36.208413 22062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:36.208514 22062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [term 1 LEADER]: Becoming Leader. State: Replica: 42271565e20d4de9a89500e1382da70d, State: Running, Role: LEADER
I20260620 09:20:36.208627 22052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:36.208675 22062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42271565e20d4de9a89500e1382da70d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38465 } }
I20260620 09:20:36.208986 22062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42271565e20d4de9a89500e1382da70d. Latest consensus state: current_term: 1 leader_uuid: "42271565e20d4de9a89500e1382da70d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42271565e20d4de9a89500e1382da70d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38465 } } }
I20260620 09:20:36.209059 22062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:36.209303 22068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:36.209844 22068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:36.211087 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:36.207828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35869 (local address 127.0.98.62:38465)
0620 09:20:36.207862 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:36.207863 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:36.207961 (+ 98us) server_negotiation.cc:407] Connection header received
0620 09:20:36.208025 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:36.208026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:36.208033 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:36.208075 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:36.208244 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:36.208899 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:36.209734 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:36.209992 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:36.210045 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:36.210797 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:36.210798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:36.210798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:36.210802 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:36.210817 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:36.210822 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:36.210865 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:36.211010 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 09:20:36.211020 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 09:20:36.211737 22063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42271565e20d4de9a89500e1382da70d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42271565e20d4de9a89500e1382da70d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38465 } } }
I20260620 09:20:36.211807 22063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:36.212401 22068 catalog_manager.cc:1347] Generated new cluster ID: ae9ecb001a104feb87156cf412993bb9
I20260620 09:20:36.212450 22068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:36.225803 22068 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:36.226327 22068 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:36.235538 22068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42271565e20d4de9a89500e1382da70d: Generated new TSK 0
I20260620 09:20:36.235793 22068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:36.238803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 21984
W20260620 09:20:36.242959 21969 connection.cc:535] client connection to 127.0.98.62:38465 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38465): Connection reset by peer (error 104)
2026-06-20T09:20:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:36 dist-test-slave-rq8c krb5kdc[22083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:36 dist-test-slave-rq8c krb5kdc[22083](info): set up 2 sockets
Jun 20 09:20:36 dist-test-slave-rq8c krb5kdc[22083](info): commencing operation
krb5kdc: starting...
W20260620 09:20:37.187489 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:37 dist-test-slave-rq8c krb5kdc[22083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:20:37.293466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44681
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:37.372254 22096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:37.372453 22096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:37.372483 22096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:37.373652 22096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:37.373711 22096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:37.373733 22096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:37.373749 22096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:37.373765 22096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:37.375224 22096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44681
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:37.375452 22096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:37.375686 22096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:37.380071 22102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:37.380249 22096 server_base.cc:1034] running on GCE node
W20260620 09:20:37.380088 22101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:37.380095 22104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:37.380597 22096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:37.380950 22096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:37.382099 22096 hybrid_clock.cc:648] HybridClock initialized: now 1781947237382079 us; error 36 us; skew 500 ppm
Jun 20 09:20:37 dist-test-slave-rq8c krb5kdc[22083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:37.384442 22096 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:20:37.384842 22096 webserver.cc:460] Webserver started at http://127.0.98.62:41943/ using document root <none> and password file <none>
I20260620 09:20:37.385047 22096 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:37.385093 22096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:37.385224 22096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:37.386261 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "726b26ad1ab7423581b4966c6a8f40d0"
format_stamp: "Formatted at 2026-06-20 09:20:37 on dist-test-slave-rq8c"
server_key: "946f66a01f42f981656780c6ac96c611"
server_key_iv: "2b7cd9af2876c541e2a9a065ff5f47fc"
server_key_version: "encryptionkey@0"
I20260620 09:20:37.386615 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "726b26ad1ab7423581b4966c6a8f40d0"
format_stamp: "Formatted at 2026-06-20 09:20:37 on dist-test-slave-rq8c"
server_key: "946f66a01f42f981656780c6ac96c611"
server_key_iv: "2b7cd9af2876c541e2a9a065ff5f47fc"
server_key_version: "encryptionkey@0"
I20260620 09:20:37.387928 22096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 09:20:37.388638 22111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:37.388789 22096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:37.388861 22096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "726b26ad1ab7423581b4966c6a8f40d0"
format_stamp: "Formatted at 2026-06-20 09:20:37 on dist-test-slave-rq8c"
server_key: "946f66a01f42f981656780c6ac96c611"
server_key_iv: "2b7cd9af2876c541e2a9a065ff5f47fc"
server_key_version: "encryptionkey@0"
I20260620 09:20:37.388919 22096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:37.392025 22096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:37.393674 22096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:37.393796 22096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:37.397411 22096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44681
I20260620 09:20:37.397450 22163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44681 every 8 connection(s)
I20260620 09:20:37.397727 22096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:37.398602 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:37.400815 22164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0: Bootstrap starting.
I20260620 09:20:37.401340 22164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:37.401576 22164 log.cc:824] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:37.402222 22164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0: No bootstrap required, opened a new log
I20260620 09:20:37.403815 22164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726b26ad1ab7423581b4966c6a8f40d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44681 } }
I20260620 09:20:37.403951 22164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:37.403972 22164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 726b26ad1ab7423581b4966c6a8f40d0, State: Initialized, Role: FOLLOWER
I20260620 09:20:37.404105 22164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726b26ad1ab7423581b4966c6a8f40d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44681 } }
I20260620 09:20:37.404177 22164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:37.404199 22164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:37.404229 22164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:37.404727 22164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726b26ad1ab7423581b4966c6a8f40d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44681 } }
I20260620 09:20:37.404815 22164 leader_election.cc:302] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 726b26ad1ab7423581b4966c6a8f40d0; no voters:
I20260620 09:20:37.404970 22164 leader_election.cc:288] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:37.405011 22167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:37.405159 22167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [term 1 LEADER]: Becoming Leader. State: Replica: 726b26ad1ab7423581b4966c6a8f40d0, State: Running, Role: LEADER
I20260620 09:20:37.405205 22164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:37.405246 22167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726b26ad1ab7423581b4966c6a8f40d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44681 } }
I20260620 09:20:37.405586 22169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 726b26ad1ab7423581b4966c6a8f40d0. Latest consensus state: current_term: 1 leader_uuid: "726b26ad1ab7423581b4966c6a8f40d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726b26ad1ab7423581b4966c6a8f40d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44681 } } }
I20260620 09:20:37.405705 22168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "726b26ad1ab7423581b4966c6a8f40d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726b26ad1ab7423581b4966c6a8f40d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44681 } } }
I20260620 09:20:37.405742 22169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:37.405784 22168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:37.406150 22176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:37.406558 22176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:37.407497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 22096
I20260620 09:20:37.407650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:37.407816 392 external_mini_cluster.cc:1428] Setting key be454c8a3568d3ab4f4daaec86bcec3b
I20260620 09:20:37.408103 22176 catalog_manager.cc:1347] Generated new cluster ID: 45c6a10441c94e03a012b385b7d221c9
I20260620 09:20:37.408149 22176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:37.410023 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:37.408424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41831 (local address 127.0.98.62:44681)
0620 09:20:37.408559 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:37.408568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:37.408595 (+ 27us) server_negotiation.cc:407] Connection header received
0620 09:20:37.408698 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:37.408704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:37.408732 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:37.408847 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:37.409627 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:37.409631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:37.409631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:37.409635 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:37.409674 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:37.409687 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:37.409744 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:37.409819 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 09:20:37.409855 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260620 09:20:37.410992 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:37.413872 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:37.412787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36699 (local address 127.0.98.62:44681)
0620 09:20:37.412838 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:37.412840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:37.413000 (+ 160us) server_negotiation.cc:407] Connection header received
0620 09:20:37.413068 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:37.413069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:37.413074 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:37.413135 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:37.413685 (+ 550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:37.413686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:37.413686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:37.413690 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:37.413704 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:37.413708 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:37.413751 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:37.413802 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 09:20:37.413813 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:20:37.422840 22176 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:37.423017 22176 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:37.435436 22176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 726b26ad1ab7423581b4966c6a8f40d0: Generated new TSK 0
I20260620 09:20:37.435669 22176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:37.438980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 22096
2026-06-20T09:20:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:37 dist-test-slave-rq8c krb5kdc[22195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:37 dist-test-slave-rq8c krb5kdc[22195](info): set up 2 sockets
Jun 20 09:20:37 dist-test-slave-rq8c krb5kdc[22195](info): commencing operation
krb5kdc: starting...
W20260620 09:20:38.383365 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:38 dist-test-slave-rq8c krb5kdc[22195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:20:38.490396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39125
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:38.567210 22208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:38.567404 22208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:38.567425 22208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:38.568584 22208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:38.568634 22208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:38.568647 22208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:38.568660 22208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:38.568670 22208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:38.570092 22208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:38.570309 22208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:38.570508 22208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:38.575176 22213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:38.575187 22216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:38.575205 22208 server_base.cc:1034] running on GCE node
W20260620 09:20:38.575168 22214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:38.575663 22208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:38.575956 22208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:38.577121 22208 hybrid_clock.cc:648] HybridClock initialized: now 1781947238577104 us; error 28 us; skew 500 ppm
Jun 20 09:20:38 dist-test-slave-rq8c krb5kdc[22195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:38.581959 22208 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:20:38.582378 22208 webserver.cc:460] Webserver started at http://127.0.98.62:42697/ using document root <none> and password file <none>
I20260620 09:20:38.582577 22208 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:38.582623 22208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:38.582739 22208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:38.583809 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "7e77455652494432aae335874de9e57f"
format_stamp: "Formatted at 2026-06-20 09:20:38 on dist-test-slave-rq8c"
server_key: "270b961161bdccafa45b896fe71ea1cc"
server_key_iv: "343b45cf2b9154e98e147a27af17f77f"
server_key_version: "encryptionkey@0"
I20260620 09:20:38.584151 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e77455652494432aae335874de9e57f"
format_stamp: "Formatted at 2026-06-20 09:20:38 on dist-test-slave-rq8c"
server_key: "270b961161bdccafa45b896fe71ea1cc"
server_key_iv: "343b45cf2b9154e98e147a27af17f77f"
server_key_version: "encryptionkey@0"
I20260620 09:20:38.585345 22208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:20:38.586014 22223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:38.586140 22208 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:38.586220 22208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "7e77455652494432aae335874de9e57f"
format_stamp: "Formatted at 2026-06-20 09:20:38 on dist-test-slave-rq8c"
server_key: "270b961161bdccafa45b896fe71ea1cc"
server_key_iv: "343b45cf2b9154e98e147a27af17f77f"
server_key_version: "encryptionkey@0"
I20260620 09:20:38.586280 22208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:38.598842 22208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:38.600560 22208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:38.600688 22208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:38.604337 22208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39125
I20260620 09:20:38.604388 22275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39125 every 8 connection(s)
I20260620 09:20:38.604717 22208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:38.605549 22276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:38.607796 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f: Bootstrap starting.
I20260620 09:20:38.608314 22276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:38.608557 22276 log.cc:824] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:38.609163 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f: No bootstrap required, opened a new log
I20260620 09:20:38.610679 22276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e77455652494432aae335874de9e57f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39125 } }
I20260620 09:20:38.610807 22276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:38.610829 22276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e77455652494432aae335874de9e57f, State: Initialized, Role: FOLLOWER
I20260620 09:20:38.610952 22276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e77455652494432aae335874de9e57f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39125 } }
I20260620 09:20:38.611011 22276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:38.611042 22276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:38.611088 22276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:38.611577 22276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e77455652494432aae335874de9e57f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39125 } }
I20260620 09:20:38.611687 22276 leader_election.cc:302] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e77455652494432aae335874de9e57f; no voters:
I20260620 09:20:38.611832 22276 leader_election.cc:288] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:38.611868 22279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:38.611996 22279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [term 1 LEADER]: Becoming Leader. State: Replica: 7e77455652494432aae335874de9e57f, State: Running, Role: LEADER
I20260620 09:20:38.612071 22276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:38.612087 22279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e77455652494432aae335874de9e57f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39125 } }
I20260620 09:20:38.612493 22281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e77455652494432aae335874de9e57f. Latest consensus state: current_term: 1 leader_uuid: "7e77455652494432aae335874de9e57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e77455652494432aae335874de9e57f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39125 } } }
I20260620 09:20:38.612489 22280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e77455652494432aae335874de9e57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e77455652494432aae335874de9e57f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39125 } } }
I20260620 09:20:38.612762 22280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:38.612668 22281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:38.612964 22287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:38.613438 22287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:38.614588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 22208
I20260620 09:20:38.614696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:38.614853 392 external_mini_cluster.cc:1428] Setting key 0d21bc3b4b97e6858e71a345cd348be6
I20260620 09:20:38.614904 22287 catalog_manager.cc:1347] Generated new cluster ID: 1ccbdd6c91f04083ba66adf3a3f20f60
I20260620 09:20:38.614943 22287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:38.619635 22287 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:38.620245 22287 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 09:20:38 dist-test-slave-rq8c krb5kdc[22195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:38.625757 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:38.615254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60511 (local address 127.0.98.62:39125)
0620 09:20:38.615507 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:38.615515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:38.615547 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:20:38.615691 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:38.615698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:38.615752 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:38.615879 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:38.616580 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:38.617999 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:38.619039 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:38.619720 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:38.619823 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:38.622463 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:38.622467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:38.622468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:38.622479 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:38.624284 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:38.624648 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:38.624650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:38.624651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:38.624741 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:38.624914 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:38.624915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:38.624915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:38.625167 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:38.625297 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:38.625390 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 09:20:38.625529 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":70,"threads_started":1}
I20260620 09:20:38.626497 22287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e77455652494432aae335874de9e57f: Generated new TSK 0
I20260620 09:20:38.626675 22287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:38.626773 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:38.637323 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:38.628030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46171 (local address 127.0.98.62:39125)
0620 09:20:38.628061 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:38.628063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:38.631765 (+ 3702us) server_negotiation.cc:407] Connection header received
0620 09:20:38.631825 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:38.631826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:38.631837 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:38.631881 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:38.632355 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:38.633084 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:38.633774 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:38.633995 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:38.634048 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:38.635155 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:38.635156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:38.635156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:38.635163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:38.636389 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:38.636673 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:38.636673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:38.636674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:38.636724 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:38.636878 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:38.636879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:38.636879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:38.636981 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:38.637060 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:38.637165 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 09:20:38.637240 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 09:20:38.639189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 22208
2026-06-20T09:20:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1200 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:38 dist-test-slave-rq8c krb5kdc[22307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:38 dist-test-slave-rq8c krb5kdc[22307](info): set up 2 sockets
Jun 20 09:20:38 dist-test-slave-rq8c krb5kdc[22307](info): commencing operation
krb5kdc: starting...
W20260620 09:20:39.579593 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:39 dist-test-slave-rq8c krb5kdc[22307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:20:39.684257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33437
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:39.761063 22320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:39.761260 22320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:39.761281 22320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:39.762415 22320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:39.762456 22320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:39.762470 22320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:39.762480 22320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:39.762503 22320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:39.764254 22320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:39.764516 22320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:39.764722 22320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:39.769587 22326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:39.769641 22325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:39.769748 22320 server_base.cc:1034] running on GCE node
W20260620 09:20:39.769644 22328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:39.770053 22320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:39.770339 22320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:39.771477 22320 hybrid_clock.cc:648] HybridClock initialized: now 1781947239771462 us; error 31 us; skew 500 ppm
Jun 20 09:20:39 dist-test-slave-rq8c krb5kdc[22307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:39.773870 22320 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:20:39.774278 22320 webserver.cc:460] Webserver started at http://127.0.98.62:40007/ using document root <none> and password file <none>
I20260620 09:20:39.774475 22320 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:39.774520 22320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:39.774641 22320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:39.775702 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "1fb67e545af1420ba75ad2dc599d31de"
format_stamp: "Formatted at 2026-06-20 09:20:39 on dist-test-slave-rq8c"
server_key: "7cc2e6b94bc8709489b259f3da475af1"
server_key_iv: "b6b2c313dc05a52c70fdb09e09c8c720"
server_key_version: "encryptionkey@0"
I20260620 09:20:39.776022 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fb67e545af1420ba75ad2dc599d31de"
format_stamp: "Formatted at 2026-06-20 09:20:39 on dist-test-slave-rq8c"
server_key: "7cc2e6b94bc8709489b259f3da475af1"
server_key_iv: "b6b2c313dc05a52c70fdb09e09c8c720"
server_key_version: "encryptionkey@0"
I20260620 09:20:39.777204 22320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:20:39.777899 22335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:39.778061 22320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:39.778141 22320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "1fb67e545af1420ba75ad2dc599d31de"
format_stamp: "Formatted at 2026-06-20 09:20:39 on dist-test-slave-rq8c"
server_key: "7cc2e6b94bc8709489b259f3da475af1"
server_key_iv: "b6b2c313dc05a52c70fdb09e09c8c720"
server_key_version: "encryptionkey@0"
I20260620 09:20:39.778199 22320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:39.795106 22320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:39.796798 22320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:39.796926 22320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:39.800455 22320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33437
I20260620 09:20:39.800489 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33437 every 8 connection(s)
I20260620 09:20:39.800788 22320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:39.801589 22388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:39.803750 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de: Bootstrap starting.
I20260620 09:20:39.804286 22388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:39.804531 22388 log.cc:824] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:39.805131 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de: No bootstrap required, opened a new log
I20260620 09:20:39.806645 22388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb67e545af1420ba75ad2dc599d31de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33437 } }
I20260620 09:20:39.806775 22388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:39.806807 22388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb67e545af1420ba75ad2dc599d31de, State: Initialized, Role: FOLLOWER
I20260620 09:20:39.806941 22388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb67e545af1420ba75ad2dc599d31de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33437 } }
I20260620 09:20:39.806999 22388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:39.807019 22388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:39.807065 22388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:39.807564 22388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb67e545af1420ba75ad2dc599d31de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33437 } }
I20260620 09:20:39.807687 22388 leader_election.cc:302] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fb67e545af1420ba75ad2dc599d31de; no voters:
I20260620 09:20:39.807837 22388 leader_election.cc:288] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:39.807881 22391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:39.808030 22391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [term 1 LEADER]: Becoming Leader. State: Replica: 1fb67e545af1420ba75ad2dc599d31de, State: Running, Role: LEADER
I20260620 09:20:39.808063 22388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:39.808116 22391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb67e545af1420ba75ad2dc599d31de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33437 } }
I20260620 09:20:39.808353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 22320
I20260620 09:20:39.808422 22392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fb67e545af1420ba75ad2dc599d31de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb67e545af1420ba75ad2dc599d31de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33437 } } }
I20260620 09:20:39.808466 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:39.808485 22392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:39.808656 392 external_mini_cluster.cc:1428] Setting key 56e8cc9361e25abea39873d9f06d70db
I20260620 09:20:39.808637 22393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fb67e545af1420ba75ad2dc599d31de. Latest consensus state: current_term: 1 leader_uuid: "1fb67e545af1420ba75ad2dc599d31de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb67e545af1420ba75ad2dc599d31de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33437 } } }
I20260620 09:20:39.808691 22393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:39.809376 22395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:39.809906 22395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:39.811450 22395 catalog_manager.cc:1347] Generated new cluster ID: b1320978acac430d970e0557ad18a1db
I20260620 09:20:39.811501 22395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:39.814297 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:39.809256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39359 (local address 127.0.98.62:33437)
0620 09:20:39.809421 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:39.809430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:39.809461 (+ 31us) server_negotiation.cc:407] Connection header received
0620 09:20:39.809581 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:39.809589 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:39.809652 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:39.809780 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:39.809966 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:39.811234 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:39.812310 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:39.812902 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:39.812989 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:39.813671 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:39.813674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:39.813674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:39.813854 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:39.813889 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:39.813903 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:39.813961 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:39.814030 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 09:20:39.814094 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260620 09:20:39.815102 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:39.819141 22395 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:39.819636 22395 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:39.823088 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:39.816572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37099 (local address 127.0.98.62:33437)
0620 09:20:39.816605 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:39.816606 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:39.819787 (+ 3181us) server_negotiation.cc:407] Connection header received
0620 09:20:39.819848 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:39.819849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:39.819860 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:39.819912 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:39.820035 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:39.820811 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:39.821821 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:39.822137 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:39.822198 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:39.822844 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:39.822845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:39.822845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:39.822850 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:39.822866 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:39.822870 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:39.822917 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:39.822974 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 09:20:39.823001 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:20:39.830751 22395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fb67e545af1420ba75ad2dc599d31de: Generated new TSK 0
I20260620 09:20:39.830919 22395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:39.832752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 22320
2026-06-20T09:20:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1195 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:39 dist-test-slave-rq8c krb5kdc[22419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:39 dist-test-slave-rq8c krb5kdc[22419](info): set up 2 sockets
Jun 20 09:20:39 dist-test-slave-rq8c krb5kdc[22419](info): commencing operation
krb5kdc: starting...
W20260620 09:20:40.783488 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:40 dist-test-slave-rq8c krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:20:40.888340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33213
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:40.965951 22432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:40.966152 22432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:40.966176 22432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:40.967311 22432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:40.967357 22432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:40.967370 22432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:40.967382 22432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:40.967406 22432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:40.968888 22432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:40.969112 22432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:40.969316 22432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:40.974020 22438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:40.974012 22437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:40.974026 22440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:40.974068 22432 server_base.cc:1034] running on GCE node
I20260620 09:20:40.974428 22432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:40.974709 22432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:40.975847 22432 hybrid_clock.cc:648] HybridClock initialized: now 1781947240975834 us; error 28 us; skew 500 ppm
Jun 20 09:20:40 dist-test-slave-rq8c krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:40.978145 22432 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:20:40.978559 22432 webserver.cc:460] Webserver started at http://127.0.98.62:39687/ using document root <none> and password file <none>
I20260620 09:20:40.978767 22432 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:40.978811 22432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:40.978917 22432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:40.979974 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "cb4b4520b102432f858b20129717a8bf"
format_stamp: "Formatted at 2026-06-20 09:20:40 on dist-test-slave-rq8c"
server_key: "1e17f370711a54dc4c2165f3672bdd44"
server_key_iv: "95191a077bae16a299163578089b2856"
server_key_version: "encryptionkey@0"
I20260620 09:20:40.980297 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "cb4b4520b102432f858b20129717a8bf"
format_stamp: "Formatted at 2026-06-20 09:20:40 on dist-test-slave-rq8c"
server_key: "1e17f370711a54dc4c2165f3672bdd44"
server_key_iv: "95191a077bae16a299163578089b2856"
server_key_version: "encryptionkey@0"
I20260620 09:20:40.981495 22432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:20:40.982194 22447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:40.982342 22432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:40.982424 22432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "cb4b4520b102432f858b20129717a8bf"
format_stamp: "Formatted at 2026-06-20 09:20:40 on dist-test-slave-rq8c"
server_key: "1e17f370711a54dc4c2165f3672bdd44"
server_key_iv: "95191a077bae16a299163578089b2856"
server_key_version: "encryptionkey@0"
I20260620 09:20:40.982483 22432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:40.985651 22432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:40.987699 22432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:40.987819 22432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:40.991293 22432 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33213
I20260620 09:20:40.991324 22499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33213 every 8 connection(s)
I20260620 09:20:40.991609 22432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:40.992254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 22432
I20260620 09:20:40.992347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:40.992496 392 external_mini_cluster.cc:1428] Setting key 343dd95a5b307ef6660b4fd94d01f76e
I20260620 09:20:40.992529 22500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:40.994733 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:40.992935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54413 (local address 127.0.98.62:33213)
0620 09:20:40.993097 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:40.993106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:40.993139 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:20:40.993672 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:40.993681 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:40.993714 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:40.993855 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:40.994353 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:40.994357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:40.994357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:40.994361 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:40.994400 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:40.994413 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:40.994486 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:40.994519 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 09:20:40.994555 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260620 09:20:40.995576 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:40.995857 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf: Bootstrap starting.
I20260620 09:20:40.996388 22500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:40.996683 22500 log.cc:824] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:40.997326 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf: No bootstrap required, opened a new log
I20260620 09:20:40.998993 22500 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4b4520b102432f858b20129717a8bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33213 } }
I20260620 09:20:40.999125 22500 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:20:40.999158 22500 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4b4520b102432f858b20129717a8bf, State: Initialized, Role: FOLLOWER
I20260620 09:20:40.999271 22500 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4b4520b102432f858b20129717a8bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33213 } }
I20260620 09:20:40.999352 22500 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:20:40.999382 22500 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:20:40.999435 22500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:41.000006 22500 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4b4520b102432f858b20129717a8bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33213 } }
I20260620 09:20:41.000114 22500 leader_election.cc:302] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb4b4520b102432f858b20129717a8bf; no voters:
I20260620 09:20:41.000277 22500 leader_election.cc:288] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:20:41.000304 22510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:41.000442 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:40.996839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45133 (local address 127.0.98.62:33213)
0620 09:20:40.996873 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:40.996874 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:40.996995 (+ 121us) server_negotiation.cc:407] Connection header received
0620 09:20:40.997058 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:40.997059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:40.997063 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:40.997092 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:41.000259 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:41.000260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:41.000261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:20:41.000265 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:41.000279 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:20:41.000283 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:41.000328 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:41.000374 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 09:20:41.000384 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:20:41.000574 22500 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:41.000473 22510 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [term 1 LEADER]: Becoming Leader. State: Replica: cb4b4520b102432f858b20129717a8bf, State: Running, Role: LEADER
I20260620 09:20:41.001871 22510 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4b4520b102432f858b20129717a8bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33213 } }
I20260620 09:20:41.002415 22512 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb4b4520b102432f858b20129717a8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4b4520b102432f858b20129717a8bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33213 } } }
I20260620 09:20:41.002409 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb4b4520b102432f858b20129717a8bf. Latest consensus state: current_term: 1 leader_uuid: "cb4b4520b102432f858b20129717a8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4b4520b102432f858b20129717a8bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33213 } } }
I20260620 09:20:41.002556 22512 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:41.002560 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf [sys.catalog]: This master's current role is: LEADER
W20260620 09:20:41.003026 22524 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:20:41.003064 22524 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:20:41.003146 22526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:41.003285 22526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:41.004774 22526 catalog_manager.cc:1347] Generated new cluster ID: 254dcf7c518b45f886c95c2f60da4859
I20260620 09:20:41.004829 22526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:41.011530 22526 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:41.011715 22526 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:41.017926 22526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb4b4520b102432f858b20129717a8bf: Generated new TSK 0
I20260620 09:20:41.018105 22526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:20:41.021611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 22432
2026-06-20T09:20:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1186 ms)
[----------] 18 tests from AuthTokenIssuingTest (21660 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:41 dist-test-slave-rq8c krb5kdc[22531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:41 dist-test-slave-rq8c krb5kdc[22531](info): set up 2 sockets
Jun 20 09:20:41 dist-test-slave-rq8c krb5kdc[22531](info): commencing operation
krb5kdc: starting...
W20260620 09:20:41.971457 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:20:42.077565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45871
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45871,127.0.98.61:40365,127.0.98.60:41125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:42.154721 22544 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:20:42.154943 22544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:42.154969 22544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:42.154994 22544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:42.156172 22544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:42.156211 22544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:42.156230 22544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:42.156246 22544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:42.156262 22544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:42.157725 22544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45871,127.0.98.61:40365,127.0.98.60:41125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:42.157967 22544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:42.158164 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:42.162590 22550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:42.162719 22552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:42.162606 22549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:42.162820 22544 server_base.cc:1034] running on GCE node
I20260620 09:20:42.163043 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:42.163298 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:42.164446 22544 hybrid_clock.cc:648] HybridClock initialized: now 1781947242164434 us; error 28 us; skew 500 ppm
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:42.167126 22544 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:20:42.167608 22544 webserver.cc:460] Webserver started at http://127.0.98.62:45269/ using document root <none> and password file <none>
I20260620 09:20:42.167872 22544 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:42.167963 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:42.168118 22544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:42.169216 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "7a4ace4214af4cb39ecbe5fd23ceea51"
format_stamp: "Formatted at 2026-06-20 09:20:42 on dist-test-slave-rq8c"
server_key: "1ef4d56f285e9715ccf594544a36fe43"
server_key_iv: "4edf584ff2b1cbae5ef258a7b0ab7842"
server_key_version: "encryptionkey@0"
I20260620 09:20:42.169541 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a4ace4214af4cb39ecbe5fd23ceea51"
format_stamp: "Formatted at 2026-06-20 09:20:42 on dist-test-slave-rq8c"
server_key: "1ef4d56f285e9715ccf594544a36fe43"
server_key_iv: "4edf584ff2b1cbae5ef258a7b0ab7842"
server_key_version: "encryptionkey@0"
I20260620 09:20:42.170743 22544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:20:42.171339 22559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:42.171478 22544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 09:20:42.171566 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "7a4ace4214af4cb39ecbe5fd23ceea51"
format_stamp: "Formatted at 2026-06-20 09:20:42 on dist-test-slave-rq8c"
server_key: "1ef4d56f285e9715ccf594544a36fe43"
server_key_iv: "4edf584ff2b1cbae5ef258a7b0ab7842"
server_key_version: "encryptionkey@0"
I20260620 09:20:42.171641 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:42.180322 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:42.181944 22544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:42.182045 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:42.185783 22544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45871
I20260620 09:20:42.185822 22611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45871 every 8 connection(s)
I20260620 09:20:42.186106 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:42.187008 22612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:42.188609 22612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:20:42.191876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 22544
I20260620 09:20:42.191979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:42.192147 392 external_mini_cluster.cc:1428] Setting key 34deff450274bd3fe6dfbe7e601cd469
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:42.198341 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.189214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45871 (local address 127.0.0.1:41276)
0620 09:20:42.189355 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.189390 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:42.189882 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:42.190116 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:42.190119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:42.190389 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:42.190526 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.190530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.191465 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.191466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.192300 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.192303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.192742 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.192743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.192804 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.193380 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:42.193390 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:42.194893 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:42.196909 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.196916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.196918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.197115 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.197368 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.197369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.197369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.197441 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.197865 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:42.197872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:42.197934 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 09:20:42.198016 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:20:42.198142 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260620 09:20:42.198359 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.189359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.0.98.62:45871)
0620 09:20:42.189455 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.189459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.189490 (+ 31us) server_negotiation.cc:407] Connection header received
0620 09:20:42.190019 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.190021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.190036 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.190108 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.190610 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.191384 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.192370 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.192655 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.192750 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.194980 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.194988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.194988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.194995 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.196821 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.197193 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.197197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.197197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.197301 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.197495 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.197496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.197496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.197768 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.197856 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.198059 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.198215 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":74,"thread_start_us":36,"threads_started":1}
I20260620 09:20:42.198982 22612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260620 09:20:42.199278 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.199176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40365 (local address 127.0.0.1:35912)
0620 09:20:42.199213 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.199261 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 09:20:42.199394 22563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40365: connect: Connection refused (error 111)
I20260620 09:20:42.199756 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.199683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40365 (local address 127.0.0.1:35914)
0620 09:20:42.199710 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.199735 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 09:20:42.199842 22612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40365: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40365: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 09:20:42.214273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40365
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45871,127.0.98.61:40365,127.0.98.60:41125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:20:42.249054 22612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } attempt: 1
I20260620 09:20:42.249547 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.249407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40365 (local address 127.0.0.1:35930)
0620 09:20:42.249477 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.249515 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 09:20:42.249917 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.249836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40365 (local address 127.0.0.1:35942)
0620 09:20:42.249871 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.249897 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 09:20:42.250010 22612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40365: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40365: connect: Connection refused (error 111)
W20260620 09:20:42.291492 22619 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:20:42.291735 22619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:42.291755 22619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:42.291774 22619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:42.292881 22619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:42.292914 22619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:42.292927 22619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:42.292938 22619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:42.292948 22619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:42.294380 22619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45871,127.0.98.61:40365,127.0.98.60:41125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:42.294565 22619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:42.294720 22619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:42.298967 22624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:42.298988 22627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:42.299121 22619 server_base.cc:1034] running on GCE node
W20260620 09:20:42.298988 22625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:42.299386 22619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:42.299661 22619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:42.300802 22619 hybrid_clock.cc:648] HybridClock initialized: now 1781947242300789 us; error 28 us; skew 500 ppm
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:42.303001 22619 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 09:20:42.303406 22619 webserver.cc:460] Webserver started at http://127.0.98.61:43971/ using document root <none> and password file <none>
I20260620 09:20:42.303593 22619 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:42.303668 22619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:42.303794 22619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:42.304761 22619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/data/instance:
uuid: "e3ddb884ad954573a621a76cec622a1a"
format_stamp: "Formatted at 2026-06-20 09:20:42 on dist-test-slave-rq8c"
server_key: "120c0bf909ec9f617ec3751bf462355e"
server_key_iv: "ae20acb46ca86983a0e4910410a737f8"
server_key_version: "encryptionkey@0"
I20260620 09:20:42.305084 22619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/wal/instance:
uuid: "e3ddb884ad954573a621a76cec622a1a"
format_stamp: "Formatted at 2026-06-20 09:20:42 on dist-test-slave-rq8c"
server_key: "120c0bf909ec9f617ec3751bf462355e"
server_key_iv: "ae20acb46ca86983a0e4910410a737f8"
server_key_version: "encryptionkey@0"
I20260620 09:20:42.306269 22619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:20:42.306989 22634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:42.307149 22619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:20:42.307227 22619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/wal
uuid: "e3ddb884ad954573a621a76cec622a1a"
format_stamp: "Formatted at 2026-06-20 09:20:42 on dist-test-slave-rq8c"
server_key: "120c0bf909ec9f617ec3751bf462355e"
server_key_iv: "ae20acb46ca86983a0e4910410a737f8"
server_key_version: "encryptionkey@0"
I20260620 09:20:42.307291 22619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:42.318166 22612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } attempt: 2
I20260620 09:20:42.318580 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.318466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40365 (local address 127.0.0.1:35946)
0620 09:20:42.318515 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.318546 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 09:20:42.318884 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.318818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40365 (local address 127.0.0.1:35962)
0620 09:20:42.318848 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.318870 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 09:20:42.318972 22612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40365: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40365: connect: Connection refused (error 111)
I20260620 09:20:42.321610 22619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:42.323244 22619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:42.323366 22619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:42.327019 22619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40365
I20260620 09:20:42.327057 22686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40365 every 8 connection(s)
I20260620 09:20:42.327359 22619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:20:42.328313 22687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:42.328310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 22619
I20260620 09:20:42.328402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-1/wal/instance
I20260620 09:20:42.328575 392 external_mini_cluster.cc:1428] Setting key 382621d323c6b54b54e95f31de481f74
I20260620 09:20:42.330189 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:42.341390 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.330818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.0.98.62:45871)
0620 09:20:42.330870 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.330873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.331030 (+ 157us) server_negotiation.cc:407] Connection header received
0620 09:20:42.331474 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.331476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.331490 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.331554 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.332162 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.332899 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.333864 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.334167 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.334240 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.338412 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.338414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.338415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.338422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.339908 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.340360 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.340361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.340361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.340427 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.340718 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.340719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.340720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.340844 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.340897 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.341236 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.341308 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 09:20:42.341588 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.330778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45871 (local address 127.0.0.1:41278)
0620 09:20:42.330924 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.330962 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:42.331392 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:42.331640 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:42.331645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:42.331893 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:42.332078 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.332082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.333002 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.333003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.333785 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.333787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.334291 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.334292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.334389 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.334836 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:42.334845 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:42.338316 (+ 3471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:42.339991 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.339998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.340000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.340267 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.340500 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.340501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.340501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.340630 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.340908 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:42.340914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:42.341075 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 09:20:42.341192 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 09:20:42.341364 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260620 09:20:42.342192 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:20:42.349411 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.342453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40365 (local address 127.0.0.1:35970)
0620 09:20:42.342508 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.342520 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:42.342563 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:42.342806 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:42.342807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:42.342946 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:42.343000 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.343002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.343891 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.343892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.344683 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.344686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.345180 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.345182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.345259 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.345588 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:42.345592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:42.346944 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:42.348485 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.348486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.348486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.348631 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.348789 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.348790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.348790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.348844 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.349120 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:42.349121 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:42.349163 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 09:20:42.349237 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:20:42.349349 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 09:20:42.349491 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.342501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.0.98.61:40365)
0620 09:20:42.342667 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.342670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.342685 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:20:42.342712 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.342714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.342725 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.342791 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.343073 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.343820 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.344773 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.345091 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.345154 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.347015 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.347021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.347022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.347028 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.348404 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.348686 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.348688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.348688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.348733 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.348895 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.348895 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.348896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.349041 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.349104 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.349266 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.349394 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260620 09:20:42.350003 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:20:42.350304 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.350209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41125 (local address 127.0.0.1:44154)
0620 09:20:42.350245 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.350287 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 09:20:42.350436 22637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 09:20:42.350819 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.350741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41125 (local address 127.0.0.1:44166)
0620 09:20:42.350775 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.350797 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 09:20:42.350911 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41125: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
I20260620 09:20:42.351215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41125
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45871,127.0.98.61:40365,127.0.98.60:41125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:20:42.400130 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } attempt: 1
I20260620 09:20:42.400656 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.400514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41125 (local address 127.0.0.1:44182)
0620 09:20:42.400571 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.400620 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 09:20:42.401057 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.400971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41125 (local address 127.0.0.1:44186)
0620 09:20:42.401010 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.401037 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 09:20:42.401158 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41125: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
I20260620 09:20:42.410115 22612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } attempt: 3
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:20:42.416502 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.410546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.0.98.61:40365)
0620 09:20:42.410622 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.410624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.410633 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:20:42.410660 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.410660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.410670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.410714 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.411025 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.411516 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.412234 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.412452 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.412506 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.414218 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.414219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.414219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.414227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.415497 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.415828 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.415829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.415830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.415877 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.416038 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.416039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.416039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.416144 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.416199 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.416328 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.416411 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 09:20:42.416503 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.410490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40365 (local address 127.0.0.1:35974)
0620 09:20:42.410546 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.410557 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:42.410598 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:42.410725 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:42.410726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:42.410887 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:42.410957 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.410960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.411598 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.411599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.412162 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.412165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.412512 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.412514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.412584 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.412988 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:42.412993 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:42.414142 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:42.415570 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.415571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.415572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.415769 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.415932 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.415933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.415933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.415991 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.416197 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:42.416198 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:42.416243 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 09:20:42.416303 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:20:42.416403 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:20:42.417071 22612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:20:42.417395 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.417302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41125 (local address 127.0.0.1:44200)
0620 09:20:42.417345 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.417375 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 09:20:42.417692 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.417621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41125 (local address 127.0.0.1:44210)
0620 09:20:42.417656 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.417677 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 09:20:42.417771 22612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41125: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
W20260620 09:20:42.430226 22694 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:20:42.430439 22694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:42.430462 22694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:42.430490 22694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:42.431641 22694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:42.431684 22694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:42.431701 22694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:42.431718 22694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:42.431740 22694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:42.433219 22694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45871,127.0.98.61:40365,127.0.98.60:41125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:42.433457 22694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:42.433650 22694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:42.438051 22702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:42.438048 22700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:42.438109 22699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:42.438191 22694 server_base.cc:1034] running on GCE node
I20260620 09:20:42.438520 22694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:42.438802 22694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:42.439934 22694 hybrid_clock.cc:648] HybridClock initialized: now 1781947242439919 us; error 26 us; skew 500 ppm
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:42.442202 22694 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 09:20:42.442571 22694 webserver.cc:460] Webserver started at http://127.0.98.60:42421/ using document root <none> and password file <none>
I20260620 09:20:42.442744 22694 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:42.442790 22694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:42.442878 22694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:42.443926 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/data/instance:
uuid: "7826d87adae140779b846c2cb75a3691"
format_stamp: "Formatted at 2026-06-20 09:20:42 on dist-test-slave-rq8c"
server_key: "11bba4be96b1e72210f29d96bf8b85a4"
server_key_iv: "eb41e372608107af2c13a48fdca9e12d"
server_key_version: "encryptionkey@0"
I20260620 09:20:42.444226 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/wal/instance:
uuid: "7826d87adae140779b846c2cb75a3691"
format_stamp: "Formatted at 2026-06-20 09:20:42 on dist-test-slave-rq8c"
server_key: "11bba4be96b1e72210f29d96bf8b85a4"
server_key_iv: "eb41e372608107af2c13a48fdca9e12d"
server_key_version: "encryptionkey@0"
I20260620 09:20:42.445374 22694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 09:20:42.445969 22709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:42.446112 22694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:20:42.446189 22694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/wal
uuid: "7826d87adae140779b846c2cb75a3691"
format_stamp: "Formatted at 2026-06-20 09:20:42 on dist-test-slave-rq8c"
server_key: "11bba4be96b1e72210f29d96bf8b85a4"
server_key_iv: "eb41e372608107af2c13a48fdca9e12d"
server_key_version: "encryptionkey@0"
I20260620 09:20:42.446242 22694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:42.448907 22612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } attempt: 1
I20260620 09:20:42.449275 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.449157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41125 (local address 127.0.0.1:44212)
0620 09:20:42.449207 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.449238 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 09:20:42.449589 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.449510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41125 (local address 127.0.0.1:44220)
0620 09:20:42.449548 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.449570 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 09:20:42.449671 22612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41125: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
I20260620 09:20:42.463857 22694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:42.465931 22694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:42.466065 22694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:42.469311 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } attempt: 2
I20260620 09:20:42.469710 22694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41125
I20260620 09:20:42.469744 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.469603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41125 (local address 127.0.0.1:44226)
0620 09:20:42.469672 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.469711 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 09:20:42.469754 22761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41125 every 8 connection(s)
I20260620 09:20:42.470180 22694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:20:42.470801 22762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:42.472419 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:20:42.475469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 22694
I20260620 09:20:42.475566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946986834672-392-0/minicluster-data/master-2/wal/instance
I20260620 09:20:42.475747 392 external_mini_cluster.cc:1428] Setting key 3b918e94bc9bcd083ad8b7bc95a1af8e
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:42.479600 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.469996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41125 (local address 127.0.0.1:44242)
0620 09:20:42.470042 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.470053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:42.470107 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:42.471350 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:42.471351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:42.471510 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:42.471560 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.471563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.472695 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.472696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.473327 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.473331 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.473965 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.473966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.474012 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.474396 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:42.474400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:42.475703 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:42.478446 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.478447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.478447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.478654 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.478899 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.478900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.478900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.478968 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.479290 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:42.479291 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:42.479349 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 09:20:42.479414 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 09:20:42.479507 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 09:20:42.479826 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.470107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.0.98.60:41125)
0620 09:20:42.470661 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.470671 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.470706 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:20:42.471158 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.471164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.471207 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.471326 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.471714 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.472608 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.473411 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.473897 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.473998 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.475794 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.475802 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.475802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.475814 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.478144 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.478734 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.478737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.478738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.478843 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.479026 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.479026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.479027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.479230 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.479289 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.479452 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.479581 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":473,"threads_started":1}
I20260620 09:20:42.480647 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.472943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45871 (local address 127.0.0.1:41288)
0620 09:20:42.473092 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.473131 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:42.473194 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:42.473332 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:42.473334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:42.473609 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:42.473668 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.473670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.474480 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.474481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.475218 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.475221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.475740 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.475742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.475820 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.476271 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:42.476275 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:42.477879 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:42.479444 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.479450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.479452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.479764 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.479988 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.479988 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.479989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.480053 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.480317 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:42.480324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:42.480374 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 09:20:42.480451 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:20:42.480547 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260620 09:20:42.481137 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.473006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41288 (local address 127.0.98.62:45871)
0620 09:20:42.473065 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.473067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.473187 (+ 120us) server_negotiation.cc:407] Connection header received
0620 09:20:42.473249 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.473249 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.473258 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.473303 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.473735 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.474398 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.475288 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.475563 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.475654 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.477966 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.477968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.477969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.477977 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.479356 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.479844 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.479845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.479845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.479905 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.480123 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.480124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.480124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.480232 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.480292 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.480971 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.481049 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:20:42.481673 22687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a: Bootstrap starting.
I20260620 09:20:42.482424 22687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:42.482667 22687 log.cc:824] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:42.483301 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:20:42.483353 22687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a: No bootstrap required, opened a new log
I20260620 09:20:42.485093 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.476064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.0.98.60:41125)
0620 09:20:42.476204 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.476206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.476516 (+ 310us) server_negotiation.cc:407] Connection header received
0620 09:20:42.476585 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.476586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.476596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.476667 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.477335 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.477949 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.479555 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.479851 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.479907 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.482095 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.482097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.482097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.482104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.484092 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.484392 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.484393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.484393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.484450 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.484603 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.484604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.484604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.484723 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.484863 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.484899 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.484972 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260620 09:20:42.485512 22687 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } }
I20260620 09:20:42.485673 22687 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ddb884ad954573a621a76cec622a1a, State: Initialized, Role: FOLLOWER
I20260620 09:20:42.485775 22687 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } }
I20260620 09:20:42.486095 22687 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:42.487545 22769 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } } }
I20260620 09:20:42.487644 22769 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 09:20:42.488097 22780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:20:42.488188 22780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:42.488394 22619 master_runner.cc:186] Error getting master registration for 127.0.98.62:45871: OK, instance_id { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" instance_seqno: 1781947242184727 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:20:42.492024 22619 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:20:42.492303 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.483525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40365 (local address 127.0.0.1:35988)
0620 09:20:42.483557 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.483567 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:42.483601 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:42.485647 (+ 2046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:42.485648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:42.485795 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:42.485853 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.485855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.486659 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.486660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.487322 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.487325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.487789 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.487790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.487860 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.488217 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:42.488222 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:42.489601 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:42.491255 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.491256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.491257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.491439 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.491651 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.491652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.491652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.491707 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.492024 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:42.492025 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:42.492068 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 09:20:42.492126 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 09:20:42.492212 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 09:20:42.492632 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.485410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.0.98.61:40365)
0620 09:20:42.485518 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.485520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.485528 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:42.485545 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.485546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.485555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.485604 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.485937 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.486593 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.487390 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.487677 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.487731 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.489693 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.489694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.489695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.489701 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.491126 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.491512 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.491513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.491513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.491566 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.491772 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.491773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.491773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.491938 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.491997 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.492479 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.492554 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 09:20:42.492925 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:20:42.494388 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.485626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.0.98.62:45871)
0620 09:20:42.485658 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.485660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.485667 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:20:42.485707 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.485708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.485717 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.485742 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.486233 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.486758 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.488759 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.489050 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.489088 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.491519 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.491521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.491521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.491528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.493079 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.493494 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.493495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.493496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.493552 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.493755 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.493756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.493757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.493889 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.494003 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.494208 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.494290 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:20:42.499380 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.493145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41125 (local address 127.0.0.1:44262)
0620 09:20:42.493189 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.493198 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:42.493231 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:42.493341 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:42.493342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:42.493463 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:42.493499 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.493500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.494067 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.494068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.494747 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.494750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.495166 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.495166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.495212 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.495532 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:42.495537 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:42.496852 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:42.498459 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.498459 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.498460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.498638 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.498804 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.498804 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.498805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.498859 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.499087 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:42.499088 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:42.499124 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 09:20:42.499184 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:20:42.499272 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 09:20:42.499424 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.493179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.0.98.60:41125)
0620 09:20:42.493220 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.493222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.493230 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:42.493278 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.493279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.493289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.493339 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.493550 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.494005 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.494821 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.495109 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.495163 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.496939 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.496940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.496941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.496947 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.498375 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.498709 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.498709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.498710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.498755 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.498912 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.498913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.498913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.499033 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.499088 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.499224 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.499320 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":21}
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:20:42.500911 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691: Bootstrap starting.
I20260620 09:20:42.501616 22762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:42.501929 22762 log.cc:824] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:42.502745 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691: No bootstrap required, opened a new log
I20260620 09:20:42.503389 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.494926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.0.98.61:40365)
0620 09:20:42.494960 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.494961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.494970 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:20:42.494989 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.494989 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.494996 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.495040 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.495584 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.496258 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.497037 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.497281 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.497334 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.500867 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.500868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.500868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.500875 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.502330 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.502661 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.502662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.502663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.502720 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.502924 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.502925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.502925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.503041 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.503169 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.503232 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.503303 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 09:20:42.504565 22762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } }
I20260620 09:20:42.504696 22762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7826d87adae140779b846c2cb75a3691, State: Initialized, Role: FOLLOWER
I20260620 09:20:42.504702 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:42.504767 22762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } }
I20260620 09:20:42.504992 22762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:42.505792 22586 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41300
W20260620 09:20:42.507876 22794 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:20:42.507930 22794 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:20:42.507411 22694 master_runner.cc:186] Error getting master registration for 127.0.98.62:45871: OK, instance_id { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" instance_seqno: 1781947242184727 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:20:42.509512 22694 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:20:42.511699 22783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } } }
I20260620 09:20:42.511798 22783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:20:42.524796 22612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } attempt: 2
Jun 20 09:20:42 dist-test-slave-rq8c krb5kdc[22531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:20:42.531875 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.525197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.0.98.60:41125)
0620 09:20:42.525253 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.525254 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.525263 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:20:42.525296 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.525297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.525305 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.525359 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.525631 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.526361 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.527018 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.527322 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.527397 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.529390 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.529391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.529391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.529398 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.530876 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.531161 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.531162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.531163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.531217 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.531404 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.531405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.531405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.531527 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.531578 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.531713 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.531785 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:20:42.531883 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.525118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41125 (local address 127.0.0.1:44264)
0620 09:20:42.525174 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:42.525186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:42.525228 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:42.525369 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:42.525370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:42.525522 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:42.525565 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.525566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.526462 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.526463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.526947 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:42.526949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.527382 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.527382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:42.527447 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.527890 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:42.527895 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:42.529297 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:42.530942 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.530943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.530944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.531101 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.531262 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:42.531263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:42.531264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:42.531339 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:42.531569 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:42.531570 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:42.531611 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 09:20:42.531692 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 09:20:42.531784 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 09:20:42.533708 22612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51: Bootstrap starting.
I20260620 09:20:42.534403 22612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:42.534691 22612 log.cc:824] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:42.535449 22612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51: No bootstrap required, opened a new log
I20260620 09:20:42.536754 22612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } }
I20260620 09:20:42.536896 22612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a4ace4214af4cb39ecbe5fd23ceea51, State: Initialized, Role: FOLLOWER
I20260620 09:20:42.536972 22612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } }
I20260620 09:20:42.537268 22612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:42.537360 22586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 09:20:42.537406 22586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:42.538125 22586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } }
I20260620 09:20:42.538375 22586 leader_election.cc:288] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [CANDIDATE]: Term 1 election: Requested vote from peers e3ddb884ad954573a621a76cec622a1a (127.0.98.61:40365), 7826d87adae140779b846c2cb75a3691 (127.0.98.60:41125)
I20260620 09:20:42.538934 22661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e3ddb884ad954573a621a76cec622a1a"
I20260620 09:20:42.539083 22661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:42.539973 22661 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a4ace4214af4cb39ecbe5fd23ceea51 in term 1.
I20260620 09:20:42.540359 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7826d87adae140779b846c2cb75a3691"
I20260620 09:20:42.540474 22736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:42.541253 22736 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a4ace4214af4cb39ecbe5fd23ceea51 in term 1.
W20260620 09:20:42.539044 22808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 09:20:42.541852 22562 leader_election.cc:302] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7826d87adae140779b846c2cb75a3691, 7a4ace4214af4cb39ecbe5fd23ceea51; no voters:
I20260620 09:20:42.544976 22544 master_runner.cc:263] No leader master found from master 7a4ace4214af4cb39ecbe5fd23ceea51
I20260620 09:20:42.545020 22544 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 09:20:42.545120 22808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:20:42.537222 22797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } } }
I20260620 09:20:42.545284 22797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:20:42.545379 22797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:42.545465 22797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [term 1 LEADER]: Becoming Leader. State: Replica: 7a4ace4214af4cb39ecbe5fd23ceea51, State: Running, Role: LEADER
I20260620 09:20:42.545579 22797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } }
I20260620 09:20:42.546072 22797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4ace4214af4cb39ecbe5fd23ceea51. Latest consensus state: current_term: 1 leader_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } } }
I20260620 09:20:42.546200 22797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:42.546448 22816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:42.547072 22816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:42.552511 22736 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [term 1 FOLLOWER]: Refusing update from remote peer 7a4ace4214af4cb39ecbe5fd23ceea51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:42.552824 22797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:42.554148 22661 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [term 1 FOLLOWER]: Refusing update from remote peer 7a4ace4214af4cb39ecbe5fd23ceea51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:42.554448 22797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:42.555562 22769 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4ace4214af4cb39ecbe5fd23ceea51. Latest consensus state: current_term: 1 leader_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } } }
I20260620 09:20:42.555671 22769 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:20:42.556314 22783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4ace4214af4cb39ecbe5fd23ceea51. Latest consensus state: current_term: 1 leader_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } } }
I20260620 09:20:42.556403 22783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:20:42.560937 22814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } } }
I20260620 09:20:42.561045 22814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:42.561475 22769 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } } }
I20260620 09:20:42.561565 22769 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:20:42.561827 22817 mvcc.cc:205] Tried to move back new op lower bound from 7298855905492815872 to 7298855905467908096. Current Snapshot: MvccSnapshot[applied={T|T < 7298855905492815872}]
I20260620 09:20:42.561928 22821 mvcc.cc:205] Tried to move back new op lower bound from 7298855905492815872 to 7298855905467908096. Current Snapshot: MvccSnapshot[applied={T|T < 7298855905492815872}]
I20260620 09:20:42.564601 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.548215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.0.98.60:41125)
0620 09:20:42.548251 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.548253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.551789 (+ 3536us) server_negotiation.cc:407] Connection header received
0620 09:20:42.551889 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.551890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.551900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.551928 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.552799 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.553551 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.555846 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.556151 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.556188 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.559736 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.559737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.559738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.559746 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.561344 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.561886 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.561888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.561888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.561948 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.563021 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.563022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.563022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.563159 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.563239 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.564433 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.564508 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:20:42.564600 22797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } } }
I20260620 09:20:42.564675 22797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:42.565068 22783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a4ace4214af4cb39ecbe5fd23ceea51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45871 } } peers { permanent_uuid: "e3ddb884ad954573a621a76cec622a1a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40365 } } peers { permanent_uuid: "7826d87adae140779b846c2cb75a3691" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41125 } } }
I20260620 09:20:42.565151 22783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:20:42.565368 22823 mvcc.cc:205] Tried to move back new op lower bound from 7298855905492815872 to 7298855905467908096. Current Snapshot: MvccSnapshot[applied={T|T < 7298855905492815872}]
I20260620 09:20:42.570792 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.548415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.0.98.61:40365)
0620 09:20:42.548449 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.548451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.554674 (+ 6223us) server_negotiation.cc:407] Connection header received
0620 09:20:42.555105 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.555107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.555117 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.555145 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.557990 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.558711 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.562102 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.562417 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.562457 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.568029 (+ 5572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.568030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.568031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.568039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.569610 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.569975 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.569976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.569977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.570028 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.570244 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.570245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.570245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.570378 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.570451 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.570626 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.570700 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:20:42.572088 22816 catalog_manager.cc:1347] Generated new cluster ID: 078d25a5ce1d4cb48630ccf2f9fddaec
I20260620 09:20:42.572134 22816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:42.577284 22816 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:42.578007 22816 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:42.590725 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:42.547997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.0.98.62:45871)
0620 09:20:42.548033 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.548035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.565658 (+ 17623us) server_negotiation.cc:407] Connection header received
0620 09:20:42.565745 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.565746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.565757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.565783 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.566278 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.567012 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.571691 (+ 4679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:42.571958 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:42.572000 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:42.587876 (+ 15876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:42.587883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:42.587883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:42.587892 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:42.589471 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.589858 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.589859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.589860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.589914 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:42.590097 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:42.590098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:42.590098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:42.590226 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:42.590325 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:42.590524 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 09:20:42.590627 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:20:42.596485 22816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a4ace4214af4cb39ecbe5fd23ceea51: Generated new TSK 0
I20260620 09:20:42.596695 22816 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:20:42.613783 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:42.612525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.0.98.61:40365)
0620 09:20:42.612568 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.612570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.613139 (+ 569us) server_negotiation.cc:407] Connection header received
0620 09:20:42.613242 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.613243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.613255 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.613283 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.613708 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36004: BlockingRecv error: recv got EOF from 127.0.0.1:36004 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 09:20:42.614674 22766 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:42.612871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.0.98.60:41125)
0620 09:20:42.612907 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.612908 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.614178 (+ 1270us) server_negotiation.cc:407] Connection header received
0620 09:20:42.614282 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.614283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.614293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.614331 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.614618 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44276: BlockingRecv error: recv got EOF from 127.0.0.1:44276 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 09:20:42.615829 22615 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:42.615121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.0.98.62:45871)
0620 09:20:42.615158 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.615160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.615273 (+ 113us) server_negotiation.cc:407] Connection header received
0620 09:20:42.615372 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.615373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.615382 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.615410 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.615766 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41310: BlockingRecv error: recv got EOF from 127.0.0.1:41310 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 09:20:42.618089 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:42.617359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.0.98.61:40365)
0620 09:20:42.617395 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.617396 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.617619 (+ 223us) server_negotiation.cc:407] Connection header received
0620 09:20:42.617716 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.617717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.617733 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.617760 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.618042 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36014: BlockingRecv error: recv got EOF from 127.0.0.1:36014 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 09:20:42.621021 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:42.620293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.0.98.61:40365)
0620 09:20:42.620336 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.620338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.620563 (+ 225us) server_negotiation.cc:407] Connection header received
0620 09:20:42.620650 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.620651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.620660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.620686 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.620970 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36016: BlockingRecv error: recv got EOF from 127.0.0.1:36016 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 09:20:42.624815 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:42.624098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.0.98.61:40365)
0620 09:20:42.624133 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.624135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.624377 (+ 242us) server_negotiation.cc:407] Connection header received
0620 09:20:42.624458 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.624459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.624469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.624496 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.624772 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36024: BlockingRecv error: recv got EOF from 127.0.0.1:36024 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 09:20:42.630754 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:42.629924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.0.98.61:40365)
0620 09:20:42.629959 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.629961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.630207 (+ 246us) server_negotiation.cc:407] Connection header received
0620 09:20:42.630326 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.630327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.630338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.630368 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.630704 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36030: BlockingRecv error: recv got EOF from 127.0.0.1:36030 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 09:20:42.640921 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:42.640034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.0.98.61:40365)
0620 09:20:42.640076 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.640078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.640347 (+ 269us) server_negotiation.cc:407] Connection header received
0620 09:20:42.640449 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.640451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.640462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.640491 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.640866 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36036: BlockingRecv error: recv got EOF from 127.0.0.1:36036 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 09:20:42.659174 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:42.658225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.0.98.61:40365)
0620 09:20:42.658268 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.658270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.658596 (+ 326us) server_negotiation.cc:407] Connection header received
0620 09:20:42.658699 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.658700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.658712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.658739 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.659115 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36050: BlockingRecv error: recv got EOF from 127.0.0.1:36050 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 09:20:42.693475 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:42.692625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.0.98.61:40365)
0620 09:20:42.692673 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.692675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.692821 (+ 146us) server_negotiation.cc:407] Connection header received
0620 09:20:42.692932 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.692933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.692947 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.693010 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.693412 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36052: BlockingRecv error: recv got EOF from 127.0.0.1:36052 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 09:20:42.759002 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:42.758382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.0.98.61:40365)
0620 09:20:42.758469 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.758471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.758532 (+ 61us) server_negotiation.cc:407] Connection header received
0620 09:20:42.758580 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.758581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.758592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.758664 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.758952 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36058: BlockingRecv error: recv got EOF from 127.0.0.1:36058 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 09:20:42.888660 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:42.888041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.0.98.61:40365)
0620 09:20:42.888109 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:42.888111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:42.888236 (+ 125us) server_negotiation.cc:407] Connection header received
0620 09:20:42.888268 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:42.888269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:42.888281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:42.888326 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:42.888598 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36060: BlockingRecv error: recv got EOF from 127.0.0.1:36060 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 09:20:43.146369 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:43.145778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.0.98.61:40365)
0620 09:20:43.145857 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:43.145863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:43.145873 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:20:43.145898 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:43.145899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:43.145909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:43.145976 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:43.146307 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36068: BlockingRecv error: recv got EOF from 127.0.0.1:36068 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:20:43.488713 22780 catalog_manager.cc:1259] Loaded cluster ID: 078d25a5ce1d4cb48630ccf2f9fddaec
I20260620 09:20:43.488795 22780 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a: loading cluster ID for follower catalog manager: success
I20260620 09:20:43.489584 22780 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a: acquiring CA information for follower catalog manager: success
I20260620 09:20:43.489878 22780 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e3ddb884ad954573a621a76cec622a1a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:20:43.508328 22794 catalog_manager.cc:1259] Loaded cluster ID: 078d25a5ce1d4cb48630ccf2f9fddaec
I20260620 09:20:43.508395 22794 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691: loading cluster ID for follower catalog manager: success
I20260620 09:20:43.509199 22794 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691: acquiring CA information for follower catalog manager: success
I20260620 09:20:43.509469 22794 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7826d87adae140779b846c2cb75a3691: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:20:43.662305 22894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:43.659598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.0.98.61:40365)
0620 09:20:43.659796 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:43.659798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:43.659810 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:20:43.659838 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:43.659838 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:43.659905 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:43.659986 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:43.660159 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:43.660670 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:43.661467 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:43.661698 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:43.661761 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:43.661897 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:43.662005 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:43.662067 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:43.662180 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 09:20:43.662206 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":107,"threads_started":1}
I20260620 09:20:43.666337 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:43.664123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44284 (local address 127.0.98.60:41125)
0620 09:20:43.664237 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:43.664238 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:43.664250 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:20:43.664284 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:43.664285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:43.664317 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:43.664383 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:43.664461 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:43.664928 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:43.665667 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:43.665868 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:43.665928 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:43.666043 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:43.666171 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:43.666233 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:43.666250 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 09:20:43.666270 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260620 09:20:43.670609 22894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:43.668046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.0.98.61:40365)
0620 09:20:43.668084 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:43.668086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:43.668094 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:43.668122 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:43.668123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:43.668132 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:43.668188 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:43.668302 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:43.668878 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:43.669871 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:43.670168 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:43.670225 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:43.670342 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:43.670373 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:43.670412 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:43.670505 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 09:20:43.670530 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:20:43.670764 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:43.668068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.0.98.60:41125)
0620 09:20:43.668101 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:43.668103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:43.668292 (+ 189us) server_negotiation.cc:407] Connection header received
0620 09:20:43.668379 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:43.668379 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:43.668390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:43.668415 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:43.668583 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:43.669286 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:43.670068 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:43.670357 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:43.670411 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:43.670533 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:43.670565 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:43.670592 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:43.670674 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 09:20:43.670693 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:20:43.671300 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:43.667955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.0.98.62:45871)
0620 09:20:43.668153 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:43.668156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:43.668163 (+ 7us) server_negotiation.cc:407] Connection header received
0620 09:20:43.668206 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:43.668207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:43.668258 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:43.668339 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:43.668422 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:43.669170 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:43.670151 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:43.670445 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:43.670503 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:43.670659 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:43.671027 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:43.671080 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:43.671145 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 09:20:43.671168 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":51,"threads_started":1}
I20260620 09:20:43.675565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 22544
I20260620 09:20:43.680124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 22619
I20260620 09:20:43.684194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 22694
2026-06-20T09:20:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2663 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:20:43 dist-test-slave-rq8c krb5kdc[22911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:20:43 dist-test-slave-rq8c krb5kdc[22911](info): set up 2 sockets
Jun 20 09:20:43 dist-test-slave-rq8c krb5kdc[22911](info): commencing operation
krb5kdc: starting...
W20260620 09:20:44.651515 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/test-user.keytab.
Jun 20 09:20:44 dist-test-slave-rq8c krb5kdc[22911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:20:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:20:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:20:44.760584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40825,127.0.98.61:42911,127.0.98.60:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:20:44.837968 22924 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:20:44.838183 22924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:44.838207 22924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:44.838230 22924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:44.839375 22924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:44.839417 22924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:44.839437 22924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:44.839453 22924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:44.839468 22924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:44.840967 22924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40825,127.0.98.61:42911,127.0.98.60:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:44.841203 22924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:44.841393 22924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:44.845759 22932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:44.845767 22929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:44.845822 22930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:44.846004 22924 server_base.cc:1034] running on GCE node
I20260620 09:20:44.846292 22924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:44.846529 22924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:44.847672 22924 hybrid_clock.cc:648] HybridClock initialized: now 1781947244847662 us; error 27 us; skew 500 ppm
Jun 20 09:20:44 dist-test-slave-rq8c krb5kdc[22911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:44.849879 22924 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:20:44.850301 22924 webserver.cc:460] Webserver started at http://127.0.98.62:38211/ using document root <none> and password file <none>
I20260620 09:20:44.850486 22924 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:44.850531 22924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:44.850646 22924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:44.851662 22924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/data/instance:
uuid: "15e75561c6424b6880ab87ebf2f7fd05"
format_stamp: "Formatted at 2026-06-20 09:20:44 on dist-test-slave-rq8c"
server_key: "884265922f42c36a23ebe1b58da4ca34"
server_key_iv: "b2efc97c6d2a5fcd4c546bc6d8da9977"
server_key_version: "encryptionkey@0"
I20260620 09:20:44.851994 22924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/wal/instance:
uuid: "15e75561c6424b6880ab87ebf2f7fd05"
format_stamp: "Formatted at 2026-06-20 09:20:44 on dist-test-slave-rq8c"
server_key: "884265922f42c36a23ebe1b58da4ca34"
server_key_iv: "b2efc97c6d2a5fcd4c546bc6d8da9977"
server_key_version: "encryptionkey@0"
I20260620 09:20:44.853160 22924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:20:44.853776 22939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:44.853919 22924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:20:44.853994 22924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/wal
uuid: "15e75561c6424b6880ab87ebf2f7fd05"
format_stamp: "Formatted at 2026-06-20 09:20:44 on dist-test-slave-rq8c"
server_key: "884265922f42c36a23ebe1b58da4ca34"
server_key_iv: "b2efc97c6d2a5fcd4c546bc6d8da9977"
server_key_version: "encryptionkey@0"
I20260620 09:20:44.854056 22924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:44.898818 22924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:44.900872 22924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:44.901026 22924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:44.904798 22924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40825
I20260620 09:20:44.904837 22991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40825 every 8 connection(s)
I20260620 09:20:44.905164 22924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:20:44.906061 22992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:44.907547 22992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:20:44 dist-test-slave-rq8c krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:44.915290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 22924
I20260620 09:20:44.915381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-0/wal/instance
I20260620 09:20:44.915570 392 external_mini_cluster.cc:1428] Setting key a2684fb80568e94009c1cb9fa78ee01e
I20260620 09:20:44.916575 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:44.908250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.0.98.62:40825)
0620 09:20:44.908374 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:44.908383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:44.908500 (+ 117us) server_negotiation.cc:407] Connection header received
0620 09:20:44.909053 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:44.909055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:44.909086 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:44.909160 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:44.909685 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:44.910510 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:44.911328 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:44.911647 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:44.911744 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:44.913642 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:44.913649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:44.913649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:44.913655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:44.915078 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:44.915517 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:44.915519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:44.915520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:44.915602 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:44.915837 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:44.915838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:44.915838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:44.916033 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:44.916102 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:44.916290 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 09:20:44.916392 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
I20260620 09:20:44.916581 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:44.908197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40825 (local address 127.0.0.1:44318)
0620 09:20:44.908366 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:44.908409 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:44.908911 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:44.909175 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:44.909177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:44.909441 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:44.909598 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:44.909602 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:44.910603 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:44.910604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:44.911254 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:44.911256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:44.911734 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:44.911736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:44.911797 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:44.912257 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:44.912265 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:44.913562 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:44.915167 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:44.915174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:44.915176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:44.915435 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:44.915674 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:44.915675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:44.915675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:44.915778 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:44.916099 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:44.916105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:44.916178 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 09:20:44.916267 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 09:20:44.916462 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":61,"thread_start_us":78,"threads_started":1}
I20260620 09:20:44.917304 22992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:20:44.917619 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:44.917513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42911 (local address 127.0.0.1:53460)
0620 09:20:44.917555 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:44.917599 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 09:20:44.917745 22942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42911: connect: Connection refused (error 111)
I20260620 09:20:44.918123 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:44.918044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42911 (local address 127.0.0.1:53466)
0620 09:20:44.918087 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:44.918109 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 09:20:44.918213 22992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42911: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42911: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 09:20:44.936542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42911
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40825,127.0.98.61:42911,127.0.98.60:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:20:44.967412 22992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } attempt: 1
I20260620 09:20:44.967924 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:44.967793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42911 (local address 127.0.0.1:53476)
0620 09:20:44.967851 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:44.967896 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 09:20:44.968271 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:44.968189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42911 (local address 127.0.0.1:53480)
0620 09:20:44.968229 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:44.968252 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 09:20:44.968354 22992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42911: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42911: connect: Connection refused (error 111)
W20260620 09:20:45.013667 22999 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:20:45.013871 22999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:45.013893 22999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:45.013916 22999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:20:45.015019 22999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:45.015057 22999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:45.015076 22999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:45.015092 22999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:45.015108 22999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:45.016538 22999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40825,127.0.98.61:42911,127.0.98.60:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:45.016762 22999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:45.016953 22999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:45.021387 23004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:45.021521 22999 server_base.cc:1034] running on GCE node
W20260620 09:20:45.021420 23005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:45.021395 23007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:45.021859 22999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:45.022130 22999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:45.023289 22999 hybrid_clock.cc:648] HybridClock initialized: now 1781947245023277 us; error 26 us; skew 500 ppm
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:45.025663 22999 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 09:20:45.026067 22999 webserver.cc:460] Webserver started at http://127.0.98.61:34341/ using document root <none> and password file <none>
I20260620 09:20:45.026249 22999 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:45.026293 22999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:45.026422 22999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:45.027421 22999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/data/instance:
uuid: "33bf0932ba4d4aa0bdfa7b45276ff228"
format_stamp: "Formatted at 2026-06-20 09:20:45 on dist-test-slave-rq8c"
server_key: "35cdd6f5cd33bd82459a9bc593aeec5e"
server_key_iv: "b3b756c2364507a5e01e8d45626b9419"
server_key_version: "encryptionkey@0"
I20260620 09:20:45.027750 22999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/wal/instance:
uuid: "33bf0932ba4d4aa0bdfa7b45276ff228"
format_stamp: "Formatted at 2026-06-20 09:20:45 on dist-test-slave-rq8c"
server_key: "35cdd6f5cd33bd82459a9bc593aeec5e"
server_key_iv: "b3b756c2364507a5e01e8d45626b9419"
server_key_version: "encryptionkey@0"
I20260620 09:20:45.028921 22999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 09:20:45.029613 23014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:45.029743 22999 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 09:20:45.029819 22999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/wal
uuid: "33bf0932ba4d4aa0bdfa7b45276ff228"
format_stamp: "Formatted at 2026-06-20 09:20:45 on dist-test-slave-rq8c"
server_key: "35cdd6f5cd33bd82459a9bc593aeec5e"
server_key_iv: "b3b756c2364507a5e01e8d45626b9419"
server_key_version: "encryptionkey@0"
I20260620 09:20:45.029876 22999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:45.036495 22992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } attempt: 2
I20260620 09:20:45.036857 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.036731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42911 (local address 127.0.0.1:53496)
0620 09:20:45.036787 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.036824 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 09:20:45.037135 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.037068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42911 (local address 127.0.0.1:53506)
0620 09:20:45.037099 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.037119 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 09:20:45.037215 22992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42911: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42911: connect: Connection refused (error 111)
I20260620 09:20:45.045675 22999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:45.047394 22999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:45.047503 22999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:45.051159 22999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42911
I20260620 09:20:45.051196 23066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42911 every 8 connection(s)
I20260620 09:20:45.051481 22999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:20:45.052385 23067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:45.053774 23067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:45.060705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 22999
I20260620 09:20:45.060796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-1/wal/instance
I20260620 09:20:45.060971 392 external_mini_cluster.cc:1428] Setting key 1fe7fcdfe71997a86fb0b1efb984c674
I20260620 09:20:45.061755 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.054379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.0.98.62:40825)
0620 09:20:45.054438 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.054440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.054606 (+ 166us) server_negotiation.cc:407] Connection header received
0620 09:20:45.055016 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.055017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.055026 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.055094 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.055612 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.056137 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.057086 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.057278 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.057327 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.059229 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.059231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.059231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.059237 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.060452 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.060884 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.060885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.060885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.060939 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.061193 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.061193 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.061194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.061299 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.061349 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.061630 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.061691 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:20:45.061957 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.054363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40825 (local address 127.0.0.1:44324)
0620 09:20:45.054506 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.054537 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:45.054949 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:45.055147 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:45.055150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:45.055370 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:45.055542 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.055546 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.056203 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.056204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.057013 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.057015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.057344 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.057345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.057458 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.057894 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:45.057902 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:45.059143 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:45.060517 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.060524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.060526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.060805 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.060998 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.060999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.060999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.061135 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.061356 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:45.061361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:45.061512 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 09:20:45.061609 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 09:20:45.061755 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260620 09:20:45.062521 23067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 09:20:45.070055 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.062743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42911 (local address 127.0.0.1:53520)
0620 09:20:45.062784 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.062793 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:45.062829 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:45.063100 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:45.063100 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:45.063205 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:45.063254 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.063255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.064085 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.064086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.064846 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.064848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.065454 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.065455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.065509 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.065968 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:45.065973 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:45.067280 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:45.069147 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.069147 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.069148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.069318 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.069474 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.069474 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.069475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.069520 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.069793 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:45.069794 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:45.069831 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 09:20:45.069891 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:20:45.069968 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 09:20:45.070122 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.062807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.0.98.61:42911)
0620 09:20:45.062955 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.062959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.062979 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:20:45.063000 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.063003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.063015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.063091 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.063309 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.064015 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.064947 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.065375 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.065457 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.067363 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.067371 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.067372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.067378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.069076 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.069381 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.069383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.069384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.069429 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.069581 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.069581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.069582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.069718 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.069778 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.069919 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.070013 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260620 09:20:45.070742 23067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 09:20:45.071036 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.070939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:46922)
0620 09:20:45.070975 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.071015 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 09:20:45.071163 23015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
I20260620 09:20:45.071534 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.071457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:46930)
0620 09:20:45.071492 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.071514 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 09:20:45.071657 23067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34237: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 09:20:45.084074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7HdOzb/build/release/bin/kudu
/tmp/dist-test-task7HdOzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34237
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40825,127.0.98.61:42911,127.0.98.60:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:20:45.120889 23067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } attempt: 1
I20260620 09:20:45.121381 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.121264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:46940)
0620 09:20:45.121321 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.121352 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 09:20:45.121832 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.121714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:46944)
0620 09:20:45.121777 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.121806 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 09:20:45.121949 23067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34237: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
I20260620 09:20:45.128364 22992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } attempt: 3
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:20:45.135479 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.128724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.0.98.61:42911)
0620 09:20:45.128790 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.128792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.128804 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:20:45.128855 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.128856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.128868 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.128923 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.129233 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.129987 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.130680 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.130998 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.131067 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.133117 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.133119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.133119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.133127 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.134459 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.134826 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.134827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.134827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.134886 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.135062 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.135062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.135062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.135165 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.135214 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.135354 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.135416 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:20:45.135538 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.128673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42911 (local address 127.0.0.1:53532)
0620 09:20:45.128740 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.128753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:45.128806 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:45.128939 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:45.128940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:45.129087 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:45.129161 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.129163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.130078 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.130078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.130609 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.130611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.131069 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.131070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.131128 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.131531 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:45.131536 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:45.132999 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:45.134538 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.134539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.134539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.134742 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.134952 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.134953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.134953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.135010 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.135217 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:45.135218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:45.135266 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 09:20:45.135337 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 09:20:45.135434 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 09:20:45.135931 22992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:20:45.136294 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.136194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:46960)
0620 09:20:45.136245 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.136273 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:20:45.136546 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.136492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:46976)
0620 09:20:45.136516 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.136534 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 09:20:45.136636 22992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34237: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
W20260620 09:20:45.166769 23074 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:20:45.166981 23074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:20:45.167012 23074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:20:45.167047 23074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 09:20:45.167778 22992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } attempt: 1
I20260620 09:20:45.168223 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.168073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:46992)
0620 09:20:45.168150 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.168193 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 09:20:45.168326 23074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:20:45.168371 23074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:20:45.168391 23074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:20:45.168414 23074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:20:45.168437 23074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:20:45.168587 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.168515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:47006)
0620 09:20:45.168543 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.168569 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 09:20:45.168674 22992 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34237: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
I20260620 09:20:45.169929 23074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40825,127.0.98.61:42911,127.0.98.60:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 08:22:15 UTC on ccycloud.quasar-qyrqrz.root.comops.site
build id 4457055
I20260620 09:20:45.170156 23074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:20:45.170369 23074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:20:45.174790 23079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:45.174786 23080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:20:45.174784 23082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:20:45.174985 23074 server_base.cc:1034] running on GCE node
I20260620 09:20:45.175328 23074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:20:45.175661 23074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:20:45.176779 23074 hybrid_clock.cc:648] HybridClock initialized: now 1781947245176761 us; error 32 us; skew 500 ppm
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:20:45.179199 23074 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 09:20:45.179598 23074 webserver.cc:460] Webserver started at http://127.0.98.60:41661/ using document root <none> and password file <none>
I20260620 09:20:45.179875 23074 fs_manager.cc:359] Metadata directory not provided
I20260620 09:20:45.179919 23074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:20:45.180009 23074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:20:45.180965 23074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/data/instance:
uuid: "789982dc59614d9bbf47f9194e7e9de6"
format_stamp: "Formatted at 2026-06-20 09:20:45 on dist-test-slave-rq8c"
server_key: "0286364ddc3be4f4b462463cc30efacc"
server_key_iv: "78e52c9088d625e60b4446c1ea855e59"
server_key_version: "encryptionkey@0"
I20260620 09:20:45.181247 23074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/wal/instance:
uuid: "789982dc59614d9bbf47f9194e7e9de6"
format_stamp: "Formatted at 2026-06-20 09:20:45 on dist-test-slave-rq8c"
server_key: "0286364ddc3be4f4b462463cc30efacc"
server_key_iv: "78e52c9088d625e60b4446c1ea855e59"
server_key_version: "encryptionkey@0"
I20260620 09:20:45.182436 23074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:20:45.183037 23089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:45.183184 23074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 09:20:45.183270 23074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/wal
uuid: "789982dc59614d9bbf47f9194e7e9de6"
format_stamp: "Formatted at 2026-06-20 09:20:45 on dist-test-slave-rq8c"
server_key: "0286364ddc3be4f4b462463cc30efacc"
server_key_iv: "78e52c9088d625e60b4446c1ea855e59"
server_key_version: "encryptionkey@0"
I20260620 09:20:45.183337 23074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:20:45.190109 23067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } attempt: 2
I20260620 09:20:45.190505 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.190395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:47008)
0620 09:20:45.190443 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.190479 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 09:20:45.190814 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.190737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:47024)
0620 09:20:45.190774 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.190797 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 09:20:45.190889 23067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34237: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34237: connect: Connection refused (error 111)
I20260620 09:20:45.199940 23074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:20:45.201823 23074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:20:45.201946 23074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:20:45.205546 23074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34237
I20260620 09:20:45.205579 23141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34237 every 8 connection(s)
I20260620 09:20:45.205886 23074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:20:45.206746 23142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:20:45.208279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7HdOzb/build/release/bin/kudu as pid 23074
I20260620 09:20:45.208374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7HdOzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946986834672-392-0/minicluster-data/master-2/wal/instance
I20260620 09:20:45.208292 23142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:20:45.208578 392 external_mini_cluster.cc:1428] Setting key 28ac1c67f611cede9e486c16e924d0e6
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:20:45.218394 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.209020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44330 (local address 127.0.98.62:40825)
0620 09:20:45.209081 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.209084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.209254 (+ 170us) server_negotiation.cc:407] Connection header received
0620 09:20:45.209728 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.209729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.209739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.209788 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.210477 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.211212 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.212429 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.212681 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.212755 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.215116 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.215117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.215118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.215126 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.217142 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.217492 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.217493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.217493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.217553 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.217770 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.217770 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.217771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.217902 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.217965 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.218222 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.218299 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 09:20:45.218570 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.208999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40825 (local address 127.0.0.1:44330)
0620 09:20:45.209139 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.209182 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:45.209636 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:45.209816 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:45.209818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:45.210183 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:45.210398 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.210404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.211298 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.211299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.212336 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.212339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.212777 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.212778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.212867 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.213339 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:45.213349 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:45.214987 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:45.217204 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.217210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.217212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.217432 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.217614 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.217615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.217615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.217715 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.217974 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:45.217979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:45.218051 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 09:20:45.218193 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 09:20:45.218352 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260620 09:20:45.218577 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.209158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.0.98.60:34237)
0620 09:20:45.209261 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.209265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.209294 (+ 29us) server_negotiation.cc:407] Connection header received
0620 09:20:45.209645 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.209651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.209669 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.209756 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.210418 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.211503 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.212525 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.212913 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.212961 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.215508 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.215516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.215516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.215523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.217106 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.217479 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.217482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.217483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.217581 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.217770 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.217770 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.217771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.218057 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.218190 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.218274 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.218441 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":77,"thread_start_us":41,"threads_started":1}
I20260620 09:20:45.219134 23142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:20:45.226007 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.219368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42911 (local address 127.0.0.1:53546)
0620 09:20:45.219412 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.219421 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:45.219454 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:45.219603 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:45.219603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:45.219764 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:45.219824 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.219826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.220699 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.220699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.221475 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.221477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.221933 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.221933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.221993 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.222405 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:45.222409 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:45.223563 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:45.225149 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.225157 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.225157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.225301 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.225461 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.225462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.225462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.225514 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.225726 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:45.225727 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:45.225769 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 09:20:45.225827 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 09:20:45.225915 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 09:20:45.226007 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.219463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.0.98.61:42911)
0620 09:20:45.219500 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.219501 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.219516 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:20:45.219537 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.219538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.219546 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.219585 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.219889 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.220618 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.221563 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.221870 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.221921 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.223674 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.223675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.223676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.223683 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.225070 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.225358 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.225359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.225359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.225411 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.225561 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.225562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.225562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.225676 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.225728 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.225853 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.225916 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:20:45.226336 23142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:20:45.226576 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.219282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.0.98.62:40825)
0620 09:20:45.219321 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.219323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.219331 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:45.219349 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.219350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.219359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.219409 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.219955 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.220646 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.221560 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.221850 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.221920 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.224238 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.224239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.224239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.224250 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.225462 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.225757 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.225758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.225758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.225807 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.225960 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.225961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.225961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.226070 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.226236 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.226415 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.226494 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:20:45.232928 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.226555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:47054)
0620 09:20:45.226756 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.226765 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:45.226811 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:45.226929 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:45.226930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:45.227056 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:45.227088 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.227090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.227920 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.227921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.228572 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.228574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.229038 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.229039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.229070 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.229365 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:45.229369 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:45.230451 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:45.231987 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.231988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.231988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.232205 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.232382 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.232382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.232383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.232435 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.232629 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:45.232630 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:45.232665 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 09:20:45.232724 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:20:45.232813 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260620 09:20:45.232985 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.226667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.0.98.60:34237)
0620 09:20:45.226716 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.226717 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.226851 (+ 134us) server_negotiation.cc:407] Connection header received
0620 09:20:45.226871 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.226871 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.226881 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.226927 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.227159 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.227863 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.228651 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.228976 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.229044 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.230541 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.230550 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.230550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.230556 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.231908 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.232276 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.232277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.232277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.232328 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.232482 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.232483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.232484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.232578 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.232630 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.232749 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.232876 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":30}
I20260620 09:20:45.234131 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.227090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.0.98.61:42911)
0620 09:20:45.227121 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.227123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.227145 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:20:45.227163 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.227163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.227172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.227210 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.227730 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.228329 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.229138 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.229376 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.229444 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.231666 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.231667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.231668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.231674 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.233082 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.233455 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.233456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.233456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.233511 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.233674 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.233675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.233675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.233778 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.233935 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.233968 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.234032 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:20:45.234483 23142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6: Bootstrap starting.
I20260620 09:20:45.235363 23142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:45.235384 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:20:45.235697 23142 log.cc:824] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:45.236433 23142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6: No bootstrap required, opened a new log
I20260620 09:20:45.236294 22966 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "15e75561c6424b6880ab87ebf2f7fd05"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44346
I20260620 09:20:45.237844 23142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } }
I20260620 09:20:45.237963 23142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789982dc59614d9bbf47f9194e7e9de6, State: Initialized, Role: FOLLOWER
I20260620 09:20:45.238037 23142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } }
I20260620 09:20:45.238338 23142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:45.238319 23148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } } }
I20260620 09:20:45.238445 23148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 09:20:45.239950 23159 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:20:45.240039 23159 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:20:45.240217 23074 master_runner.cc:186] Error getting master registration for 127.0.98.62:40825: OK, instance_id { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" instance_seqno: 1781947244903726 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:20:45.242179 23074 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:20:45.243759 22992 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } attempt: 2
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:20:45.250339 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.244063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.0.98.60:34237)
0620 09:20:45.244099 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.244101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.244110 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:20:45.244173 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.244174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.244181 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.244233 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.244499 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.244957 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.245603 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.245818 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.245872 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.247666 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.247668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.247668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.247675 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.249189 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.249578 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.249579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.249579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.249631 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.249800 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.249800 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.249801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.249963 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.250024 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.250173 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.250247 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 09:20:45.250349 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.244010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:47056)
0620 09:20:45.244068 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.244079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:45.244117 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:45.244251 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:45.244252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:45.244399 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:45.244437 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.244439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.245022 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.245023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.245526 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.245528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.245875 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.245876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.245930 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.246328 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:45.246333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:45.247549 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:45.249276 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.249278 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.249278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.249505 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.249683 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.249684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.249684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.249746 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.250022 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:45.250023 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:45.250080 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 09:20:45.250148 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 09:20:45.250249 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 09:20:45.252094 22992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05: Bootstrap starting.
I20260620 09:20:45.252671 22992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:45.252959 22992 log.cc:824] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:45.253744 22992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05: No bootstrap required, opened a new log
I20260620 09:20:45.254726 22992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } }
I20260620 09:20:45.254837 22992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e75561c6424b6880ab87ebf2f7fd05, State: Initialized, Role: FOLLOWER
I20260620 09:20:45.254909 22992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } }
I20260620 09:20:45.255158 23162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } } }
I20260620 09:20:45.255219 22992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:45.255237 23162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:20:45.255337 22966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 09:20:45.255409 22966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:20:45.256268 22966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } }
I20260620 09:20:45.256955 23116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15e75561c6424b6880ab87ebf2f7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "789982dc59614d9bbf47f9194e7e9de6"
I20260620 09:20:45.257093 23116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [term 0 FOLLOWER]: Advancing to term 1
W20260620 09:20:45.257242 23173 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:20:45.257287 23173 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:20:45.257915 23116 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e75561c6424b6880ab87ebf2f7fd05 in term 1.
I20260620 09:20:45.258136 22940 leader_election.cc:302] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e75561c6424b6880ab87ebf2f7fd05, 789982dc59614d9bbf47f9194e7e9de6; no voters:
I20260620 09:20:45.258255 23162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:20:45.258368 23162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [term 1 LEADER]: Becoming Leader. State: Replica: 15e75561c6424b6880ab87ebf2f7fd05, State: Running, Role: LEADER
I20260620 09:20:45.258438 23162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } }
I20260620 09:20:45.258797 23162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15e75561c6424b6880ab87ebf2f7fd05. Latest consensus state: current_term: 1 leader_uuid: "15e75561c6424b6880ab87ebf2f7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } } }
I20260620 09:20:45.258886 23162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:45.259177 22966 leader_election.cc:288] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [CANDIDATE]: Term 1 election: Requested vote from peers 33bf0932ba4d4aa0bdfa7b45276ff228 (127.0.98.61:42911), 789982dc59614d9bbf47f9194e7e9de6 (127.0.98.60:34237)
I20260620 09:20:45.259871 23177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:20:45.257864 22924 master_runner.cc:186] Error getting master registration for 127.0.98.61:42911: OK, instance_id { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" instance_seqno: 1781947245050118 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:20:45.260070 22924 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:20:45.260198 23177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:20:45.268448 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.260971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.0.98.62:40825)
0620 09:20:45.261012 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.261014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.261172 (+ 158us) server_negotiation.cc:407] Connection header received
0620 09:20:45.261264 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.261265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.261275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.261302 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.262019 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.262756 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.263819 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.264132 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.264170 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.265677 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.265678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.265679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.265685 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.267236 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.267600 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.267601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.267602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.267676 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.267869 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.267869 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.267870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.267986 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.268086 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.268294 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.268365 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 09:20:45.268488 23116 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [term 1 FOLLOWER]: Refusing update from remote peer 15e75561c6424b6880ab87ebf2f7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:45.268935 23175 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:45.270219 23148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15e75561c6424b6880ab87ebf2f7fd05. Latest consensus state: current_term: 1 leader_uuid: "15e75561c6424b6880ab87ebf2f7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } } }
I20260620 09:20:45.270316 23148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:20:45.272516 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15e75561c6424b6880ab87ebf2f7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } } }
I20260620 09:20:45.272614 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:45.273053 23148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15e75561c6424b6880ab87ebf2f7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } } }
I20260620 09:20:45.273132 23148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:20:45.273805 23186 mvcc.cc:205] Tried to move back new op lower bound from 7298855916617510912 to 7298855916579307520. Current Snapshot: MvccSnapshot[applied={T|T < 7298855916617510912}]
I20260620 09:20:45.274809 23177 catalog_manager.cc:1347] Generated new cluster ID: 4425c27afdb843918e2fd56dd068b3a4
I20260620 09:20:45.274854 23177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:20:45.277141 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.261191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.0.98.61:42911)
0620 09:20:45.261232 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.261234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.268965 (+ 7731us) server_negotiation.cc:407] Connection header received
0620 09:20:45.269070 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.269071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.269081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.269109 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.269785 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.270512 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.271551 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.271877 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.271918 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.273432 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.273433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.273434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.273440 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.275329 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.275797 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.275798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.275798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.275856 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.276301 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.276302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.276302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.276671 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.276759 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.276959 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.277033 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 09:20:45.278206 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.261535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.0.98.60:34237)
0620 09:20:45.261584 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.261587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.271767 (+ 10180us) server_negotiation.cc:407] Connection header received
0620 09:20:45.271859 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.271859 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.271869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.271895 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.272593 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.273223 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.274190 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.274459 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.274516 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.275787 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.275789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.275789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.275795 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.277194 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.277514 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.277514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.277515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.277557 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.277708 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.277709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.277709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.277817 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.277879 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.278059 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.278125 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 09:20:45.281952 23067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } attempt: 3
Jun 20 09:20:45 dist-test-slave-rq8c krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781947245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:20:45.290591 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.282196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34237 (local address 127.0.0.1:47078)
0620 09:20:45.282273 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 09:20:45.282283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 09:20:45.282317 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:20:45.282531 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:20:45.282532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:20:45.282689 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:20:45.282729 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.282730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.283716 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.283717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.284508 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:20:45.284510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.285014 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.285015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:20:45.285092 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.285623 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:20:45.285628 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:20:45.287168 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:20:45.289480 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.289481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.289481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.289689 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.289913 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:20:45.289914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:20:45.289914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:20:45.289982 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:20:45.290288 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:20:45.290289 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:20:45.290346 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 09:20:45.290402 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 09:20:45.290491 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 09:20:45.291149 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:45.282379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.0.98.60:34237)
0620 09:20:45.282421 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.282423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.282429 (+ 6us) server_negotiation.cc:407] Connection header received
0620 09:20:45.282447 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.282448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.282456 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.282496 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.282847 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.283572 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.284606 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:45.284894 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:45.284948 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:45.287268 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:20:45.287270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:20:45.287270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:20:45.287276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:20:45.289383 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.289773 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.289774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.289774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.289839 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:20:45.290062 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:20:45.290063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:20:45.290063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:20:45.290196 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:20:45.290249 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:45.290984 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 09:20:45.291064 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 09:20:45.292608 23067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228: Bootstrap starting.
I20260620 09:20:45.293303 23067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228: Neither blocks nor log segments found. Creating new log.
I20260620 09:20:45.293602 23067 log.cc:824] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228: Log is configured to *not* fsync() on all Append() calls
I20260620 09:20:45.294576 23067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228: No bootstrap required, opened a new log
I20260620 09:20:45.294955 23177 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:20:45.295655 23177 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:20:45.296959 23067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } }
I20260620 09:20:45.297086 23067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33bf0932ba4d4aa0bdfa7b45276ff228, State: Initialized, Role: FOLLOWER
I20260620 09:20:45.297165 23067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } }
I20260620 09:20:45.297452 23067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:20:45.297544 23040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [term 0 FOLLOWER]: Advancing to term 1
W20260620 09:20:45.298082 23020 tablet.cc:2369] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:20:45.298384 23040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [term 1 FOLLOWER]: Refusing update from remote peer 15e75561c6424b6880ab87ebf2f7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:20:45.298826 23175 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:20:45.298542 23041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15e75561c6424b6880ab87ebf2f7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228"
I20260620 09:20:45.300160 23041 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e75561c6424b6880ab87ebf2f7fd05 in term 1.
I20260620 09:20:45.300346 23177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05: Generated new TSK 0
I20260620 09:20:45.300505 23177 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:20:45.302983 23202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:20:45.303032 23202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:20:45.297453 23191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } } }
I20260620 09:20:45.303262 23191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:20:45.303339 23191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15e75561c6424b6880ab87ebf2f7fd05. Latest consensus state: current_term: 1 leader_uuid: "15e75561c6424b6880ab87ebf2f7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } } }
I20260620 09:20:45.303427 23191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:20:45.303488 23191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15e75561c6424b6880ab87ebf2f7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } } }
I20260620 09:20:45.303685 23191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:20:45.312484 23162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15e75561c6424b6880ab87ebf2f7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e75561c6424b6880ab87ebf2f7fd05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40825 } } peers { permanent_uuid: "33bf0932ba4d4aa0bdfa7b45276ff228" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42911 } } peers { permanent_uuid: "789982dc59614d9bbf47f9194e7e9de6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34237 } } }
I20260620 09:20:45.312611 23162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15e75561c6424b6880ab87ebf2f7fd05 [sys.catalog]: This master's current role is: LEADER
I20260620 09:20:45.312947 23204 mvcc.cc:205] Tried to move back new op lower bound from 7298855916702924800 to 7298855916579307520. Current Snapshot: MvccSnapshot[applied={T|T < 7298855916617510912}]
W20260620 09:20:45.328383 22995 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.327014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.0.98.62:40825)
0620 09:20:45.327062 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.327064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.327732 (+ 668us) server_negotiation.cc:407] Connection header received
0620 09:20:45.327835 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.327836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.327848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.327877 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.328311 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44372: BlockingRecv error: recv got EOF from 127.0.0.1:44372 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 09:20:45.329088 23071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.327402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.0.98.61:42911)
0620 09:20:45.327439 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.327441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.328621 (+ 1180us) server_negotiation.cc:407] Connection header received
0620 09:20:45.328710 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.328711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.328721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.328749 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.329027 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53580: BlockingRecv error: recv got EOF from 127.0.0.1:53580 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 09:20:45.330077 23146 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.329440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47084 (local address 127.0.98.60:34237)
0620 09:20:45.329497 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.329499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.329592 (+ 93us) server_negotiation.cc:407] Connection header received
0620 09:20:45.329677 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.329677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.329687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.329714 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.330027 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47084: BlockingRecv error: recv got EOF from 127.0.0.1:47084 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 09:20:45.332260 23071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.331461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.0.98.61:42911)
0620 09:20:45.331496 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.331498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.331741 (+ 243us) server_negotiation.cc:407] Connection header received
0620 09:20:45.331837 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.331838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.331853 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.331880 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.332199 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53592: BlockingRecv error: recv got EOF from 127.0.0.1:53592 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 09:20:45.335096 23071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.334377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53600 (local address 127.0.98.61:42911)
0620 09:20:45.334408 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.334410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.334658 (+ 248us) server_negotiation.cc:407] Connection header received
0620 09:20:45.334750 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.334751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.334759 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.334786 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.335053 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53600: BlockingRecv error: recv got EOF from 127.0.0.1:53600 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 09:20:45.338798 23071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.338073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53606 (local address 127.0.98.61:42911)
0620 09:20:45.338104 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.338105 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.338319 (+ 214us) server_negotiation.cc:407] Connection header received
0620 09:20:45.338437 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.338438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.338447 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.338473 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.338757 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53606: BlockingRecv error: recv got EOF from 127.0.0.1:53606 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260620 09:20:45.344571 23071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.343812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.0.98.61:42911)
0620 09:20:45.343848 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.343849 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.344086 (+ 237us) server_negotiation.cc:407] Connection header received
0620 09:20:45.344175 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.344176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.344185 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.344210 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.344515 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53614: BlockingRecv error: recv got EOF from 127.0.0.1:53614 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 09:20:45.354704 23071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.353718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.0.98.61:42911)
0620 09:20:45.353825 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.353826 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.354092 (+ 266us) server_negotiation.cc:407] Connection header received
0620 09:20:45.354196 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.354197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.354208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.354237 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.354635 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53630: BlockingRecv error: recv got EOF from 127.0.0.1:53630 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 09:20:45.373214 23071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.372317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53632 (local address 127.0.98.61:42911)
0620 09:20:45.372359 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.372360 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.372640 (+ 280us) server_negotiation.cc:407] Connection header received
0620 09:20:45.372745 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.372746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.372758 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.372788 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.373155 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53632: BlockingRecv error: recv got EOF from 127.0.0.1:53632 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 09:20:45.407368 23071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.406451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.0.98.61:42911)
0620 09:20:45.406506 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.406508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.406780 (+ 272us) server_negotiation.cc:407] Connection header received
0620 09:20:45.406876 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.406878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.406890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.406918 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.407304 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53640: BlockingRecv error: recv got EOF from 127.0.0.1:53640 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 09:20:45.473202 23071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.472622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.0.98.61:42911)
0620 09:20:45.472697 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.472699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.472755 (+ 56us) server_negotiation.cc:407] Connection header received
0620 09:20:45.472806 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.472807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.472817 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.472862 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.473144 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53652: BlockingRecv error: recv got EOF from 127.0.0.1:53652 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 09:20:45.602898 23071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.602229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.0.98.61:42911)
0620 09:20:45.602299 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.602301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.602368 (+ 67us) server_negotiation.cc:407] Connection header received
0620 09:20:45.602424 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.602425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.602436 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.602485 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.602837 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53666: BlockingRecv error: recv got EOF from 127.0.0.1:53666 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 09:20:45.860675 23071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:20:45.859975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.0.98.61:42911)
0620 09:20:45.860054 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:45.860056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:45.860071 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:20:45.860097 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:45.860098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:45.860108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:45.860193 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:20:45.860614 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34394: BlockingRecv error: recv got EOF from 127.0.0.1:34394 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:20:46.240545 23159 catalog_manager.cc:1259] Loaded cluster ID: 4425c27afdb843918e2fd56dd068b3a4
I20260620 09:20:46.240623 23159 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6: loading cluster ID for follower catalog manager: success
I20260620 09:20:46.241372 23159 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6: acquiring CA information for follower catalog manager: success
I20260620 09:20:46.241626 23159 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 789982dc59614d9bbf47f9194e7e9de6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:20:46.303495 23202 catalog_manager.cc:1259] Loaded cluster ID: 4425c27afdb843918e2fd56dd068b3a4
I20260620 09:20:46.303579 23202 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228: loading cluster ID for follower catalog manager: success
I20260620 09:20:46.304384 23202 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228: acquiring CA information for follower catalog manager: success
I20260620 09:20:46.304646 23202 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 33bf0932ba4d4aa0bdfa7b45276ff228: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:20:46.375968 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:46.373707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.0.98.61:42911)
0620 09:20:46.373832 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:46.373834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:46.373845 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:20:46.373906 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:46.373907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:46.373957 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:46.374033 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:46.374151 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:46.374643 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:46.375344 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:46.375525 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:46.375567 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:46.375691 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:46.375793 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:46.375846 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:46.375865 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 09:20:46.375882 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260620 09:20:46.379983 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:46.377674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.0.98.60:34237)
0620 09:20:46.377797 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:46.377799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:46.377807 (+ 8us) server_negotiation.cc:407] Connection header received
0620 09:20:46.377842 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:46.377842 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:46.377875 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:46.377941 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:46.378019 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:46.378492 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:46.379218 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:46.379429 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:46.379486 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:46.379656 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:46.379777 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:46.379838 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:46.379873 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 09:20:46.379897 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260620 09:20:46.384109 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:46.381551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.0.98.61:42911)
0620 09:20:46.381644 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:46.381646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:46.381788 (+ 142us) server_negotiation.cc:407] Connection header received
0620 09:20:46.381873 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:46.381875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:46.381884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:46.381935 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:46.382024 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:46.382510 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:46.383402 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:46.383711 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:46.383773 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:46.383877 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:46.383910 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:46.383952 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:46.384012 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 09:20:46.384034 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 09:20:46.384409 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:46.381604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.0.98.60:34237)
0620 09:20:46.381674 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:46.381676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:46.381788 (+ 112us) server_negotiation.cc:407] Connection header received
0620 09:20:46.381902 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:46.381903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:46.381912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:46.381949 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:46.382087 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:46.382807 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:46.383652 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:46.383917 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:46.383972 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:46.384152 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:46.384184 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:46.384211 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:46.384318 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 09:20:46.384341 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:20:46.384641 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:20:46.381553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.0.98.62:40825)
0620 09:20:46.381724 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 09:20:46.381725 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 09:20:46.381778 (+ 53us) server_negotiation.cc:407] Connection header received
0620 09:20:46.381859 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:20:46.381860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:20:46.381910 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:20:46.381977 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:20:46.382153 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:46.382892 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:46.383735 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:20:46.384025 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:20:46.384078 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:20:46.384290 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:20:46.384399 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:20:46.384449 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:20:46.384531 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 09:20:46.384558 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":44,"threads_started":1}
I20260620 09:20:46.386389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 22924
I20260620 09:20:46.391294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 22999
I20260620 09:20:46.395443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7HdOzb/build/release/bin/kudu with pid 23074
2026-06-20T09:20:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2711 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5374 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (259565 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST